Error come while excute > es-sync sample.yaml command.

[NikhilTank35]

I setup all things correctly but some how it give this

[INFO] - init.py[line:468] - 2020-09-16 02:04:27,010 Start to dump from stream
[INFO] - init.py[line:475] - 2020-09-16 02:05:44,098 Dump success
[INFO] - init.py[line:489] - 2020-09-16 02:05:44,099 Start to sync binlog
[ERROR] - init.py[line:510] - 2020-09-16 02:05:44,104 Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/es_sync/init.py", line 507, in run
self._sync_from_binlog()
File "/usr/local/lib/python3.6/site-packages/es_sync/init.py", line 493, in _sync_from_binlog
self._updater(docs)
File "/usr/local/lib/python3.6/site-packages/es_sync/init.py", line 176, in _updater
for item in data:
File "/usr/local/lib/python3.6/site-packages/es_sync/init.py", line 202, in _processor
for item in data:
File "/usr/local/lib/python3.6/site-packages/es_sync/init.py", line 233, in _mapper
for item in data:
File "/usr/local/lib/python3.6/site-packages/es_sync/init.py", line 283, in _binlog_loader
for binlogevent in stream:
File "/usr/local/lib/python3.6/site-packages/pymysqlreplication/binlogstream.py", line 361, in fetchone
self.__connect_to_stream()
File "/usr/local/lib/python3.6/site-packages/pymysqlreplication/binlogstream.py", line 268, in __connect_to_stream
cur.execute("SHOW MASTER STATUS")
File "/usr/local/lib/python3.6/site-packages/pymysql/cursors.py", line 146, in execute
result = self._query(query)
File "/usr/local/lib/python3.6/site-packages/pymysql/cursors.py", line 296, in _query
conn.query(q)
File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 781, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 942, in _read_query_result
result.read()
File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 1138, in read
first_packet = self.connection._read_packet()
File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 906, in _read_packet
packet.check_error()
File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 367, in check_error
err.raise_mysql_exception(self._data)
File "/usr/local/lib/python3.6/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
_check_mysql_exception(errinfo)
File "/usr/local/lib/python3.6/site-packages/pymysql/err.py", line 115, in _check_mysql_exception
raise InternalError(errno, errorvalue)
pymysql.err.InternalError: (1227, 'Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation')

anyone know why this error comes !!

[NikhilTank35]

@Windfarer

[NikhilTank35]

@gogogen

[NikhilTank35]

@huangweicong112

[Windfarer]

you should grant needed privileges to your user as your error log says:

pymysql.err.InternalError: (1227, 'Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation')

[NikhilTank35]

@Windfarer Thanks for your reply . I created super user for same and it solve this error.

[NikhilTank35]

@Windfarer

But now i am getting this error,

[INFO] - init.py[line:468] - 2020-09-16 05:14:28,964 Start to dump from stream
[INFO] - init.py[line:475] - 2020-09-16 05:15:45,032 Dump success
[INFO] - init.py[line:489] - 2020-09-16 05:15:45,032 Start to sync binlog
[ERROR] - init.py[line:510] - 2020-09-16 05:15:45,039 Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/es_sync/init.py", line 507, in run
self._sync_from_binlog()
File "/usr/local/lib/python3.6/site-packages/es_sync/init.py", line 493, in _sync_from_binlog
self._updater(docs)
File "/usr/local/lib/python3.6/site-packages/es_sync/init.py", line 176, in _updater
for item in data:
File "/usr/local/lib/python3.6/site-packages/es_sync/init.py", line 202, in _processor
for item in data:
File "/usr/local/lib/python3.6/site-packages/es_sync/init.py", line 233, in _mapper
for item in data:
File "/usr/local/lib/python3.6/site-packages/es_sync/init.py", line 283, in _binlog_loader
for binlogevent in stream:
File "/usr/local/lib/python3.6/site-packages/pymysqlreplication/binlogstream.py", line 361, in fetchone
self.__connect_to_stream()
File "/usr/local/lib/python3.6/site-packages/pymysqlreplication/binlogstream.py", line 269, in __connect_to_stream
self.log_file, self.log_pos = cur.fetchone()[:2]
TypeError: 'NoneType' object is not subscriptable

do you or someone know why this error occurred ?

[Windfarer]

check your binlog is enabled

[NikhilTank35]

@Windfarer Thanks for your reply. I enabled binlog and again execute command but some how it stop working in between . i observe in mylog.py file.
let me share you log output

mylog.py
[INFO] - init.py[line:410] - 2020-09-17 13:36:10,572 Sync binlog_file: mysql-bin.000002 binlog_pos: 85249230
[INFO] - init.py[line:410] - 2020-09-17 13:36:10,873 Sync binlog_file: mysql-bin.000002 binlog_pos: 85328914
[INFO] - init.py[line:410] - 2020-09-17 13:36:10,936 Sync binlog_file: mysql-bin.000002 binlog_pos: 85408716
[INFO] - init.py[line:410] - 2020-09-17 13:36:10,997 Sync binlog_file: mysql-bin.000002 binlog_pos: 85488482

binlog.info
log_file: mysql-bin.000002
log_pos: 85488482

It stop executing now.
Do you or someone know why this stop executing ?

[Windfarer]

It is watching your binlog update, and keeping mysql and elasticsearch data synchronized, you could modify your mysql table data to see if the modification can be synchronized.

[iexpertini]

It is watching your binlog update, and keeping mysql and elasticsearch data synchronized, you could modify your mysql table data to see if the modification can be synchronized.

Hi
Me and Nikhil is working together.

Thanks for your reply but we noted in Elasticsearch there is no index is created yet.

[Windfarer]

I leaved this company long time ago, and have no permission to maintain this project. Since this project hasn't been maintained many years, maybe cannot compatible with the new version of Elasticsearch, you could also try to use other tool instead, or fork and modify this project by your own.