At this time this PR appears to contain many more commits than actually changed; this is because there's still other outstanding requests to merge upstream. For reference, the specific changes to this PR are https://github.com/openark/gh-ost/pull/13/files

Resubmission of https://github.com/openark/gh-ost/pull/13 from downstream.

Supporting the case where any unique key, and in particular the PRIMARY KEY, includes a GENERATED column.

A user with a large database deployment, who initially reported the issue, has confirmed the fix works as expected.

• [x] contributed code is using same conventions as original code
• [x] script/cibuild returns with no formatting errors, build errors or unit test errors.

Origin table does not alter with --execute

Script Info

#!/usr/bin/env bash
set -eo pipefail

DATABASE_USER=xxxx
DATABASE_PASSWORD="xxxx"
DATABASE_HOST=10.0.0.113
DATABASE_PORT=3306
DATABASE="test"
TABLE="issues"

./gh-ost --aliyun-rds \
--host="$DATABASE_HOST" \
--port="$DATABASE_PORT" \
--user="$DATABASE_USER" \
--password="$DATABASE_PASSWORD" \
--database="$DATABASE" \
--table="$TABLE" \
--alter="add COLUMN country_code varchar(32) COMMENT 'National Code'" \
--allow-on-master \
--verbose \
--debug \
--ok-to-drop-table \
--initially-drop-ghost-table \
--initially-drop-socket-file \
--panic-flag-file=/tmp/ghost.panic.flag \
--throttle-flag-file=/tmp/1.log \
–-assume-rbr=true \
--execute

Debug message

2021-01-11 16:58:15 INFO starting gh-ost 1.1.0
2021-01-11 16:58:15 INFO Migrating `test`.`issues`
2021-01-11 16:58:15 INFO connection validated on 10.0.0.113:3306
2021-01-11 16:58:15 INFO User has REPLICATION CLIENT, REPLICATION SLAVE privileges, and has ALL privileges on `test`.*
2021-01-11 16:58:15 INFO binary logs validated on 10.0.0.113:3306
2021-01-11 16:58:15 INFO Restarting replication on 10.0.0.113:3306 to make sure binlog settings apply to replication thread
2021-01-11 16:58:15 INFO Inspector initiated on 10.0.0.113:3306, version 10.1.9-MariaDB-V2.0R500D002-20190826-1611
2021-01-11 16:58:15 INFO Table found. Engine=InnoDB
2021-01-11 16:58:15 DEBUG Estimated number of rows via STATUS: 3188103
2021-01-11 16:58:15 DEBUG Validated no foreign keys exist on table
2021-01-11 16:58:15 DEBUG Validated no triggers exist on table
2021-01-11 16:58:15 INFO Estimated number of rows via EXPLAIN: 3188103
2021-01-11 16:58:15 DEBUG Potential unique keys in issues: [PRIMARY (auto_increment): [id]; has nullable: false]
2021-01-11 16:58:15 INFO Recursively searching for replication master
2021-01-11 16:58:15 DEBUG Looking for master on 10.0.0.113:3306
2021-01-11 16:58:15 INFO Master found to be 10.0.0.113:3306
2021-01-11 16:58:15 INFO log_slave_updates validated on 10.0.0.113:3306
2021-01-11 16:58:15 INFO connection validated on 10.0.0.113:3306
2021-01-11 16:58:15 DEBUG Streamer binlog coordinates: binlog.001941:37007317
2021-01-11 16:58:15 INFO Connecting binlog streamer at binlog.001941:37007317
[2021/01/11 16:58:15] [info] binlogsyncer.go:133 create BinlogSyncer with config {99999 mysql 10.0.0.113 3306 ops    false false <nil> false UTC true 0 0s 0s 0 false}
[2021/01/11 16:58:15] [info] binlogsyncer.go:354 begin to sync binlog from position (binlog.001941, 37007317)
[2021/01/11 16:58:15] [info] binlogsyncer.go:203 register slave for master server 10.0.0.113:3306
2021-01-11 16:58:15 DEBUG Beginning streaming
2021-01-11 16:58:15 INFO rotate to next log from binlog.001941:0 to binlog.001941
[2021/01/11 16:58:15] [info] binlogsyncer.go:723 rotate to (binlog.001941, 37007317)
2021-01-11 16:58:15 INFO connection validated on 10.0.0.113:3306
2021-01-11 16:58:15 INFO connection validated on 10.0.0.113:3306
2021-01-11 16:58:15 INFO will use time_zone='SYSTEM' on applier
2021-01-11 16:58:15 INFO Examining table structure on applier
2021-01-11 16:58:15 INFO Applier initiated on 10.0.0.113:3306, version 10.1.9-MariaDB-V2.0R500D002-20190826-1611
2021-01-11 16:58:15 INFO Dropping table `test`.`_issues_gho`
2021-01-11 16:58:15 INFO Table dropped
2021-01-11 16:58:15 INFO Dropping table `test`.`_issues_ghc`
2021-01-11 16:58:15 INFO Table dropped
2021-01-11 16:58:15 INFO Creating changelog table `test`.`_issues_ghc`
2021-01-11 16:58:15 INFO Changelog table created
2021-01-11 16:58:15 INFO Creating ghost table `test`.`_issues_gho`
2021-01-11 16:58:15 INFO Ghost table created
2021-01-11 16:58:15 INFO Altering ghost table `test`.`_issues_gho`
2021-01-11 16:58:15 DEBUG ALTER statement: alter /* gh-ost */ table `test`.`_issues_gho` add COLUMN country_code varchar(32) COMMENT 'National Code'
2021-01-11 16:58:15 INFO Ghost table altered
2021-01-11 16:58:15 INFO Intercepted changelog state GhostTableMigrated
2021-01-11 16:58:15 INFO Waiting for ghost table to be migrated. Current lag is 0s
2021-01-11 16:58:15 DEBUG ghost table migrated
2021-01-11 16:58:15 INFO Handled changelog state GhostTableMigrated
2021-01-11 16:58:15 DEBUG Potential unique keys in _issues_gho: [PRIMARY (auto_increment): [id]; has nullable: false]
2021-01-11 16:58:15 INFO Chosen shared unique key is PRIMARY
2021-01-11 16:58:15 INFO Shared columns are id,project_id,parent_id,parent_type,code,type,name,description,iteration_id,join_iteration_at,issue_status_id,issue_status_name,issue_status_type,priority,assignee,start_date,due_date,working_hours,creator,status_updated_at,todo_started_at,completed_at,sub_task_sort,story_point,created_at,updated_at,deleted_at,team_id
2021-01-11 16:58:15 INFO Listening on unix socket file: /tmp/gh-ost.test.issues.sock
2021-01-11 16:58:15 DEBUG Reading migration range according to key: PRIMARY
2021-01-11 16:58:15 INFO Migration min values: [5]
2021-01-11 16:58:15 DEBUG Reading migration range according to key: PRIMARY
2021-01-11 16:58:15 INFO Migration max values: [10923463]
2021-01-11 16:58:15 INFO Waiting for first throttle metrics to be collected
2021-01-11 16:58:15 INFO First throttle metrics collected
2021-01-11 16:58:15 DEBUG Operating until row copy is complete
# Migrating `test`.`issues`; Ghost table is `test`.`_issues_gho`
2021-01-11 16:58:15 DEBUG Noop operation; not really executing write funcs
2021-01-11 16:58:15 DEBUG Noop operation; not really copying data
2021-01-11 16:58:15 INFO Row copy complete
# Migrating `test`.`issues`; Ghost table is `test`.`_issues_gho`
# Migrating 10.0.0.113:3306; inspecting 10.0.0.113:3306; executing on staging-gateway
# Migration started at Mon Jan 11 16:58:15 +0800 2021
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-flag-file: /tmp/1.log
# throttle-additional-flag-file: /tmp/gh-ost.throttle
# panic-flag-file: /tmp/ghost.panic.flag
# Serving on unix socket: /tmp/gh-ost.test.issues.sock
# Migrating 10.0.0.113:3306; inspecting 10.0.0.113:3306; executing on staging-gateway
# Migration started at Mon Jan 11 16:58:15 +0800 2021
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-flag-file: /tmp/1.log
# throttle-additional-flag-file: /tmp/gh-ost.throttle
# panic-flag-file: /tmp/ghost.panic.flag
# Serving on unix socket: /tmp/gh-ost.test.issues.sock
Copy: 0/0 100.0%; Applied: 0; Backlog: 0/1000; Time: 0s(total), 0s(copy); streamer: binlog.001941:37009311; Lag: 0.02s, State: migrating; ETA: due
2021-01-11 16:58:15 DEBUG Noop operation; not really swapping tables
2021-01-11 16:58:15 INFO New table structure follows
CREATE TABLE `_issues_gho` (
xxx
xxx
xxx
xxx
xxx
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
[2021/01/11 16:58:15] [info] binlogsyncer.go:164 syncer is closing...
[2021/01/11 16:58:15] [error] binlogstreamer.go:77 close sync with err: sync is been closing...
2021-01-11 16:58:15 INFO Closed streamer connection. err=<nil>
[2021/01/11 16:58:15] [info] binlogsyncer.go:179 syncer is closed
2021-01-11 16:58:15 INFO Dropping table `test`.`_issues_ghc`
2021-01-11 16:58:15 DEBUG Done streaming
Copy: 0/3188103 0.0%; Applied: 0; Backlog: 0/1000; Time: 0s(total), 0s(copy); streamer: binlog.001941:37009311; Lag: 0.02s, State: migrating; ETA: N/A
2021-01-11 16:58:15 INFO Table dropped
2021-01-11 16:58:15 INFO Dropping table `test`.`_issues_del`
2021-01-11 16:58:15 INFO Table dropped
2021-01-11 16:58:15 INFO Dropping table `test`.`_issues_gho`
2021-01-11 16:58:15 INFO Table dropped
2021-01-11 16:58:15 INFO Done migrating `test`.`issues`
2021-01-11 16:58:15 INFO Removing socket file: /tmp/gh-ost.test.issues.sock
2021-01-11 16:58:15 INFO Tearing down inspector
2021-01-11 16:58:15 INFO Tearing down applier
2021-01-11 16:58:15 DEBUG Tearing down...
2021-01-11 16:58:15 INFO Tearing down streamer
2021-01-11 16:58:15 INFO Tearing down throttler
2021-01-11 16:58:15 DEBUG Tearing down...
# Done

Additional

We use Aliyun-rds based on MariaDB 5.6 and the schema of temp table _issues_gho is correctly transformed, Sry I have to desensitize it. After the script is done and I checked the origin table... Nothing changed, any tips or solution ?

ghost version: 1.1.0

My bad. It seems is --execute is unused in the script. It is a usage mistake of the shell script. Close