How to execute a pig script in apache Airflow? I am trying to execute a pig script but getting error

This is my pig script

data =load 'textfile' using PigStorage('\n') as (line:chararray);
dump data;

 

 

This is the airflow dag script PigOperator configurations:

pig_script= open('/home/cloudera/pig_script.pig').read()
t4 = PigOperator(
task_id= 'pig_job',
pig_cli_conn_id= 'hive_conn_inet',
pig =pig_script,
pigparams_jinja_translate=True,
depends_on_past=False,
dag=dag
)

 

This is the error i am getting :

[2017-06-28 06:55:15,047] {models.py:1219} INFO - Executing <Task(PigOperator): pig_job> on 2017-06-28 00:00:00
[2017-06-28 06:55:15,093] {pig_operator.py:50} INFO - Executing: data =load 'textfile' using PigStorage('\n') as (line:chararray);
dump data;
[2017-06-28 06:55:15,116] {base_hook.py:53} INFO - Using connection to: 10.20.174.2
[2017-06-28 06:55:15,124] {models.py:1286} ERROR - a bytes-like object is required, not 'str'
Traceback (most recent call last):
File "/root/anaconda3/lib/python3.5/site-packages/airflow/models.py", line 1245, in run
result = task_copy.execute(context=context)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/operators/pig_operator.py", line 52, in execute
self.hook.run_cli(pig=self.pig)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/hooks/pig_hook.py", line 41, in run_cli
f.write(pig)
File "/root/anaconda3/lib/python3.5/tempfile.py", line 483, in func_wrapper
return func(*args, **kwargs)
TypeError: a bytes-like object is required, not 'str'
[2017-06-28 06:55:15,132] {models.py:1298} INFO - Marking task as UP_FOR_RETRY
[2017-06-28 06:55:15,133] {models.py:1327} ERROR - a bytes-like object is required, not 'str'
Traceback (most recent call last):
File "/root/anaconda3/bin/airflow", line 15, in <module>
args.func(args)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/bin/cli.py", line 352, in test
ti.run(force=True, ignore_dependencies=True, test_mode=True)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/utils/db.py", line 53, in wrapper
result = func(*args, **kwargs)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/models.py", line 1245, in run
result = task_copy.execute(context=context)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/operators/pig_operator.py", line 52, in execute
self.hook.run_cli(pig=self.pig)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/hooks/pig_hook.py", line 41, in run_cli
f.write(pig)
File "/root/anaconda3/lib/python3.5/tempfile.py", line 483, in func_wrapper
return func(*args, **kwargs)
TypeError: a bytes-like object is required, not 'str'

 

 

 

1 answer

2 votes

I think you've landed in the wrong place.  You'll need to explain what this has to do with Atlassian products if you have not.

Suggest an answer

Log in or Sign up to answer
Atlassian Community Anniversary

Happy Anniversary, Atlassian Community!

This community is celebrating its one-year anniversary and Atlassian co-founder Mike Cannon-Brookes has all the feels.

Read more
Community showcase
Kesha Thillainayagam
Posted Apr 13, 2018 in Confluence

We want to hear how your non-technical teams are using Confluence!

Hi Community! Kesha (kay-sha) from the Confluence marketing team here! Can you share stories with us on how your non-technical (think Marketing, Sales, HR, legal, etc.) teams are using Confluen...

387 views 21 10
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you