This is my pig script
data =load 'textfile' using PigStorage('\n') as (line:chararray);
dump data;
This is the airflow dag script PigOperator configurations:
pig_script= open('/home/cloudera/pig_script.pig').read()
t4 = PigOperator(
task_id= 'pig_job',
pig_cli_conn_id= 'hive_conn_inet',
pig =pig_script,
pigparams_jinja_translate=True,
depends_on_past=False,
dag=dag
)
This is the error i am getting :
[2017-06-28 06:55:15,047] {models.py:1219} INFO - Executing <Task(PigOperator): pig_job> on 2017-06-28 00:00:00
[2017-06-28 06:55:15,093] {pig_operator.py:50} INFO - Executing: data =load 'textfile' using PigStorage('\n') as (line:chararray);
dump data;
[2017-06-28 06:55:15,116] {base_hook.py:53} INFO - Using connection to: 10.20.174.2
[2017-06-28 06:55:15,124] {models.py:1286} ERROR - a bytes-like object is required, not 'str'
Traceback (most recent call last):
File "/root/anaconda3/lib/python3.5/site-packages/airflow/models.py", line 1245, in run
result = task_copy.execute(context=context)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/operators/pig_operator.py", line 52, in execute
self.hook.run_cli(pig=self.pig)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/hooks/pig_hook.py", line 41, in run_cli
f.write(pig)
File "/root/anaconda3/lib/python3.5/tempfile.py", line 483, in func_wrapper
return func(*args, **kwargs)
TypeError: a bytes-like object is required, not 'str'
[2017-06-28 06:55:15,132] {models.py:1298} INFO - Marking task as UP_FOR_RETRY
[2017-06-28 06:55:15,133] {models.py:1327} ERROR - a bytes-like object is required, not 'str'
Traceback (most recent call last):
File "/root/anaconda3/bin/airflow", line 15, in <module>
args.func(args)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/bin/cli.py", line 352, in test
ti.run(force=True, ignore_dependencies=True, test_mode=True)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/utils/db.py", line 53, in wrapper
result = func(*args, **kwargs)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/models.py", line 1245, in run
result = task_copy.execute(context=context)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/operators/pig_operator.py", line 52, in execute
self.hook.run_cli(pig=self.pig)
File "/root/anaconda3/lib/python3.5/site-packages/airflow/hooks/pig_hook.py", line 41, in run_cli
f.write(pig)
File "/root/anaconda3/lib/python3.5/tempfile.py", line 483, in func_wrapper
return func(*args, **kwargs)
TypeError: a bytes-like object is required, not 'str'
I think you've landed in the wrong place. You'll need to explain what this has to do with Atlassian products if you have not.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.