Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

How to execute a pig script in apache Airflow? I am trying to execute a pig script but getting error

parool singh June 28, 2017

This is my pig script

data =load 'textfile' using PigStorage('\n') as (line:chararray);
dump data;

 

 

This is the airflow dag script PigOperator configurations:

pig_script= open('/home/cloudera/pig_script.pig').read()
t4 = PigOperator(
task_id= 'pig_job',
pig_cli_conn_id= 'hive_conn_inet',
pig =pig_script,
pigparams_jinja_translate=True,
depends_on_past=False,
dag=dag
)

 

This is the error i am getting :

[2017-06-28 06:55:15,047] {models.py:1219} INFO - Executing <Task(PigOperator): pig_job> on 2017-06-28 00:00:00
[2017-06-28 06:55:15,093] {pig_operator.py:50} INFO - Executing: data =load 'textfile' using PigStorage('\n') as (line:chararray);
dump data;
[2017-06-28 06:55:15,116] {base_hook.py:53} INFO - Using connection to: 10.20.174.2
[2017-06-28 06:55:15,124] {models.py:1286} ERROR - a bytes-like object is required, not 'str'
Traceback (most recent call last):
File "/root/anaconda3/lib/python3.5/site-packages/airflow/models.py", line 1245, in run
result = task_copy.execute(context=context)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/operators/pig_operator.py", line 52, in execute
self.hook.run_cli(pig=self.pig)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/hooks/pig_hook.py", line 41, in run_cli
f.write(pig)
File "/root/anaconda3/lib/python3.5/tempfile.py", line 483, in func_wrapper
return func(*args, **kwargs)
TypeError: a bytes-like object is required, not 'str'
[2017-06-28 06:55:15,132] {models.py:1298} INFO - Marking task as UP_FOR_RETRY
[2017-06-28 06:55:15,133] {models.py:1327} ERROR - a bytes-like object is required, not 'str'
Traceback (most recent call last):
File "/root/anaconda3/bin/airflow", line 15, in <module>
args.func(args)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/bin/cli.py", line 352, in test
ti.run(force=True, ignore_dependencies=True, test_mode=True)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/utils/db.py", line 53, in wrapper
result = func(*args, **kwargs)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/models.py", line 1245, in run
result = task_copy.execute(context=context)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/operators/pig_operator.py", line 52, in execute
self.hook.run_cli(pig=self.pig)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/hooks/pig_hook.py", line 41, in run_cli
f.write(pig)
File "/root/anaconda3/lib/python3.5/tempfile.py", line 483, in func_wrapper
return func(*args, **kwargs)
TypeError: a bytes-like object is required, not 'str'

 

 

 

1 answer

2 votes
Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
June 28, 2017

I think you've landed in the wrong place.  You'll need to explain what this has to do with Atlassian products if you have not.

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events