Goldengate的一大卖点就是可以实现在线的数据迁移，这意味着在整个过程中我们无需关闭数据库，甚至于无需停止应用。在 中，大致介绍了实现零下线时间数据迁移所涉及的步骤，包括:
- 在source与target主机上完成OGG的初始化配置
- 创建extract及exttrail
- 创建pump
- 启动本地extract和pump
- 创建带有HANDLECOLLISIONS参数的replicat，但不启动它
- 完成数据的initial load即数据初始化
- 启动replicat
- 当replicat追上extract后取消replicat的HANDLECOLLISIONS参数
SQL> conn clinic/clinicConnected./* 以clinic为应用用户 */SQL> create table tv (t1 int primary key,t2 int,t3 varchar2(30));Table created./* 为别在source和target创建该测试用表,在这里我们先不考虑同步DDL*/SQL> create sequence seqt1 start with 1 increment by 1;Sequence created.declare rnd number(9,2);begin for i in 1..100000 loop insert into tv values(seqt1.nextval,i*dbms_random.value,'MACLEAN IS TESTING'); commit; end loop;end;PL/SQL procedure successfully completed./* 在源端随机生成10w行数据 */begin loop delete from tv where rownum=1; commit; insert into tv values(seqt1.nextval,200000*dbms_random.value,'MACLEAN IS UPDATING'); commit; insert into tv values(seqt1.nextval,300000*dbms_random.value,'MACLEAN IS UPDATING'); commit; update tv set t2=t2+10 where rownum=1; commit; dbms_lock.sleep(1); end loop;end;/* 以上匿名块可以造成简易的数据变化，它会在每1秒钟删除/更新/插入TV表的部分数据 我们打开多个窗口执行以上匿名块，以模拟在线业务*/以上为测试前提工作，接下来我们正式配置OGG FOR ZERO DOWNTIME MIGRATION:
GGSCI (rh2.oracle.com) 3> dblogin userid macleanPassword:Successfully logged into database.GGSCI (rh2.oracle.com) 5> add trandata clinic.tvLogging of supplemental redo data enabled for table CLINIC.TV./* 通过add trandata命令为tv表加上追加日志 */GGSCI (rh2.oracle.com) 6> add extract load2,tranlog,begin nowEXTRACT added.GGSCI (rh2.oracle.com) 7> add rmttrail /s01/new/ze,megabytes 200,extract load2RMTTRAIL added.GGSCI (rh2.oracle.com) 10> view params load2extract load2userid maclean, password AACAAAAAAAAAAAHANCUEFHPAVCYDNJVD,encryptkey defaultRMTHOST rh3.oracle.com, MGRPORT 7809RMTTRAIL /s01/new/zeTable clinic.tv;/* 为了让整个过程更简洁，我们在这里不适用pump */GGSCI (rh2.oracle.com) 9> start load2Sending START request to MANAGER ...EXTRACT LOAD2 startingGGGSCI (rh2.oracle.com) 11> stats load2Sending STATS request to EXTRACT LOAD2 ...Start of Statistics at 2010-12-11 20:00:44.Output to /s01/new/ze:Extracting from CLINIC.TV to CLINIC.TV:*** Total statistics since 2010-12-11 20:00:32 *** Total inserts 370.00 Total updates 185.00 Total deletes 185.00 Total discards 0.00 Total operations 740.00*** Daily statistics since 2010-12-11 20:00:32 *** Total inserts 370.00 Total updates 185.00 Total deletes 185.00 Total discards 0.00 Total operations 740.00*** Hourly statistics since 2010-12-11 20:00:32 *** Total inserts 370.00 Total updates 185.00 Total deletes 185.00 Total discards 0.00 Total operations 740.00*** Latest statistics since 2010-12-11 20:00:32 *** Total inserts 370.00 Total updates 185.00 Total deletes 185.00 Total discards 0.00 Total operations 740.00End of Statistics./* 以上完成了extract的配置和启动，接下来我们配置target上的replicat */GGSCI (rh3.oracle.com) 39> add replicat rep2,begin now,exttrail /s01/new/ze,checkpointtable maclean.checkpointREPLICAT added.GGSCI (rh3.oracle.com) 42> view params rep2replicat rep2userid maclean,password macleanASSUMETARGETDEFSdiscardfile /s01/discard/clinic.log,append,megabytes 10HANDLECOLLISIONSmap clinic.tv, target clinic.tv;/* 注意为replicat加上HANDLECOLLISIONS参数， 用以处理可能的deplicate records和missed records; 该参数的具体定义见Reference文档*//* 注意在配置完replicat后，不要去启动它；直到完成initial load才能启动*//* 接下来要配置initial load使用的extract和replicat */GGSCI (rh2.oracle.com) 14> add extract einit,sourceistableEXTRACT added./* 以sourceistable选项创建extract */GGSCI (rh2.oracle.com) 17> view params einitextract einitSOURCEISTABLEuserid maclean, password macleanRMTHOST rh3.oracle.com, MGRPORT 7809RMTFILE /s01/new/clinictvTable clinic.tv;/* 之后我们需要在命令行界面下调用该extract */[maclean@rh2 gg]$ pwd/home/maclean/gg[maclean@rh2 gg]$ extract paramfile dirprm/einit.prm2010-12-11 20:03:06 INFO OGG-01017 Wildcard resolution set to IMMEDIATE because SOURCEISTABLE is used.*********************************************************************** Oracle GoldenGate Capture for Oracle Version 11.1.1.0.0 Build 078 Linux, x64, 64bit (optimized), Oracle 10 on Jul 28 2010 15:39:00 Copyright (C) 1995, 2010, Oracle and/or its affiliates. All rights reserved. Starting at 2010-12-11 20:03:06***********************************************************************Operating System Version:LinuxVersion #1 SMP Mon Mar 29 22:10:29 EDT 2010, Release 2.6.18-194.el5Node: rh2.oracle.comMachine: x86_64 soft limit hard limitAddress Space Size : unlimited unlimitedHeap Size : unlimited unlimitedFile Size : unlimited unlimitedCPU Time : unlimited unlimitedProcess id: 21399Description:************************************************************************* Running with the following parameters *************************************************************************extract einitSOURCEISTABLE2010-12-11 20:03:06 INFO OGG-01017 Wildcard resolution set to IMMEDIATE because SOURCEISTABLE is used.userid maclean, password *******RMTHOST rh3.oracle.com, MGRPORT 7809RMTFILE /s01/new/clinictvTable clinic.tv;Using the following key columns for source table CLINIC.TV: T1.CACHEMGR virtual memory values (may have been adjusted)CACHEBUFFERSIZE: 64KCACHESIZE: 8GCACHEBUFFERSIZE (soft max): 4MCACHEPAGEOUTSIZE (normal): 4MPROCESS VM AVAIL FROM OS (min): 16GCACHESIZEMAX (strict force to disk): 13.99GDatabase Version:Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64biPL/SQL Release 10.2.0.4.0 - ProductionCORE 10.2.0.4.0 ProductionTNS for Linux: Version 10.2.0.4.0 - ProductionNLSRTL Version 10.2.0.4.0 - ProductionDatabase Language and Character Set:NLS_LANG environment variable specified has invalid format, default value will be used.NLS_LANG environment variable not set, using default value AMERICAN_AMERICA.US7ASCII.NLS_LANGUAGE = "AMERICAN"NLS_TERRITORY = "AMERICA"NLS_CHARACTERSET = "WE8ISO8859P1"Warning: your NLS_LANG setting does not match database server language setting.Please refer to user manual for more information.2010-12-11 20:03:06 INFO OGG-01478 Output file /s01/new/clinictv is using format RELEASE 10.4/11.1.2010-12-11 20:03:11 INFO OGG-01226 Socket buffer size set to 27985 (flush size 27985).Processing table CLINIC.TV************************************************************************ ** Run Time Statistics ** ************************************************************************Report at 2010-12-11 20:03:12 (activity since 2010-12-11 20:03:06)Output to /s01/new/clinictv:From Table CLINIC.TV: # inserts: 100780 # updates: 0 # deletes: 0 # discards: 0/* 以上extract操作在target上RMTFILE指定的路径生成了导出文件*/[maclean@rh3 new]$ pwd/s01/new[maclean@rh3 new]$ file clinictvclinictv: data[maclean@rh3 new]$ strings clinictv |head -20uri:rh2:oracle:com:6/s01/new/clinictv7Linux1rh2.oracle.com22.6.18-194.el53##1 SMP Mon Mar 29 22:10:29 EDT 20104x86_642CLINICA2clinica3Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64biPL/SQL Release 10.2.0.4.0 - ProductionCORE 10.2.0.4.0 ProductionTNS for Linux: Version 10.2.0.4.0 - ProductionNLSRTL Version 10.2.0.4.0 - ProductionEINIT1Version 11.1.1.0.0 Build 0784CLINIC.TVMACLEAN IS TESTINGZCLINIC.TVMACLEAN IS TESTINGZ/* 接下来我们要在target上配置initial load使用的replicat，以导入之前的EXTFILE */GGSCI (rh3.oracle.com) 12> add replicat rinit,specialrunREPLICAT added.GGSCI (rh3.oracle.com) 17> view params rinitreplicat rinitSPECIALRUNuserid maclean,password macleanEXTFILE /s01/new/clinictvASSUMETARGETDEFSmap clinic.tv, target clinic.tv;/ *之后我们在命令行上调用该replicat */[maclean@rh3 gg]$ replicat paramfile dirprm/rinit.prm*********************************************************************** Oracle GoldenGate Delivery for Oracle Version 11.1.1.0.0 Build 078 Linux, x64, 64bit (optimized), Oracle 10 on Jul 28 2010 15:58:11Copyright (C) 1995, 2010, Oracle and/or its affiliates. All rights reserved. Starting at 2010-12-11 20:06:14***********************************************************************Operating System Version:LinuxVersion #1 SMP Mon Mar 29 22:22:00 EDT 2010, Release 2.6.18-194.el5xenNode: rh3.oracle.comMachine: x86_64 soft limit hard limitAddress Space Size : unlimited unlimitedHeap Size : unlimited unlimitedFile Size : unlimited unlimitedCPU Time : unlimited unlimitedProcess id: 18554Description:************************************************************************* Running with the following parameters *************************************************************************replicat rinitSPECIALRUNuserid maclean,password *******EXTFILE /s01/new/clinictvASSUMETARGETDEFSmap clinic.tv, target clinic.tv;CACHEMGR virtual memory values (may have been adjusted)CACHEBUFFERSIZE: 64KCACHESIZE: 512MCACHEBUFFERSIZE (soft max): 4MCACHEPAGEOUTSIZE (normal): 4MPROCESS VM AVAIL FROM OS (min): 1GCACHESIZEMAX (strict force to disk): 881MDatabase Version:Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64biPL/SQL Release 10.2.0.4.0 - ProductionCORE 10.2.0.4.0 ProductionTNS for Linux: Version 10.2.0.4.0 - ProductionNLSRTL Version 10.2.0.4.0 - ProductionDatabase Language and Character Set:NLS_LANG environment variable specified has invalid format, default value will be used.NLS_LANG environment variable not set, using default value AMERICAN_AMERICA.US7ASCII.NLS_LANGUAGE = "AMERICAN"NLS_TERRITORY = "AMERICA"NLS_CHARACTERSET = "WE8ISO8859P1"Warning: your NLS_LANG setting does not match database server language setting.Please refer to user manual for more information.************************************************************************* Run Time Messages *************************************************************************Opened trail file /s01/new/clinictv at 2010-12-11 20:06:14MAP resolved (entry CLINIC.TV): map CLINIC.TV, target clinic.tv;Using following columns in default map by name: T1, T2, T3Using the following key columns for target table CLINIC.TV: T1.************************************************************************ ** Run Time Statistics ** ************************************************************************Last record for the last committed transaction is the following:___________________________________________________________________Trail name : /s01/new/clinictvHdr-Ind : E (x45) Partition : . (x04)UndoFlag : . (x00) BeforeAfter: A (x41)RecLength : 52 (x0034) IO Time : 2010-12-11 20:03:10.969955IOType : 5 (x05) OrigNode : 255 (xff)TransInd : . (x03) FormatType : R (x52)SyskeyLen : 0 (x00) Incomplete : . (x00)AuditRBA : 0 AuditPos : 0Continued : N (x00) RecCount : 1 (x01)2010-12-11 20:03:10.969955 Insert Len 52 RBA 11244940Name: CLINIC.TV___________________________________________________________________Reading /s01/new/clinictv, current RBA 11245052, 100780 recordsReport at 2010-12-11 20:06:54 (activity since 2010-12-11 20:06:14)From Table CLINIC.TV to CLINIC.TV: # inserts: 100780 # updates: 0 # deletes: 0 # discards: 0Last log location read: FILE: /s01/new/clinictv RBA: 11245052 TIMESTAMP: 2010-12-11 20:03:10.969955 EOF: NO READERR: 400SQL> select count(*) from tv; COUNT(*)---------- 100780/* 以上完成了initial load所要求的工作，接着我们打开同步使用的replicat*/GGSCI (rh3.oracle.com) 3> start rep2Sending START request to MANAGER ...REPLICAT REP2 startingGGSCI (rh3.oracle.com) 7> lag rep2Sending GETLAG request to REPLICAT REP2 ...Last record lag: 4 seconds.At EOF, no more records to process.GGSCI (rh3.oracle.com) 9> stats rep2Sending STATS request to REPLICAT REP2 ...Start of Statistics at 2010-12-11 20:10:15.Replicating from CLINIC.TV to CLINIC.TV:*** Total statistics since 2010-12-11 20:09:54 *** Total inserts 204.00 Total updates 102.00 Total deletes 102.00 Total discards 0.00 Total operations 408.00*** Daily statistics since 2010-12-11 20:09:54 *** Total inserts 204.00 Total updates 102.00 Total deletes 102.00 Total discards 0.00 Total operations 408.00*** Hourly statistics since 2010-12-11 20:09:54 *** Total inserts 204.00 Total updates 102.00 Total deletes 102.00 Total discards 0.00 Total operations 408.00*** Latest statistics since 2010-12-11 20:09:54 *** Total inserts 204.00 Total updates 102.00 Total deletes 102.00 Total discards 0.00 Total operations 408.00End of Statistics./* 当replicat追上extract后，需要取消HANDLECOLLISIONS参数 */GGSCI (rh3.oracle.com) 11> view params rep2replicat rep2userid maclean,password macleanASSUMETARGETDEFSdiscardfile /s01/discard/clinic.log,append,megabytes 10map clinic.tv, target clinic.tv;GGSCI (rh3.oracle.com) 12> stop rep2Sending STOP request to REPLICAT REP2 ...Request processed.GGSCI (rh3.oracle.com) 13> start rep2Sending START request to MANAGER ...REPLICAT REP2 starting接下来停止模拟的在线业务，进行数据对比。首先使用Goldengate veridata创建group及compare pair等配置: 接着生成job，并执行: Job运行的结果界面: 对比发现同步正常的报告:
*********************************************************************** Oracle GoldenGate Veridata Server Version 3.0.0.6 Build 002Copyright (C) 2004, 2009, Oracle and/or its affiliates. All rights reserved. Starting at 2010-12-11 20:15:58***********************************************************************Unexpected Windows major versionProcess id: 6340, Thread id : 6392Group: clinic (1) Compare PairsPerforming 1 comparisons ... Compare pair: TV=TV2010-12-11 20:15:58. Submitted (1 of 1) Compare Pair (TV=TV) RunId (runid=(1007, 1, 1)).2010-12-11 20:16:00. Completed in-sync. Compare Pair (TV=TV). (0) Remaining Compare Pairs with Errors: 0 Compare Pairs with Out-Of-Sync: 0 Compare Pairs without Out-Of-Sync: 1 Compare Pairs Cancelled: 0Group terminated normally.接着我们在target端执行一些人工修改数据的SQL，以达到不同步的效果:
begin loop delete from tv where rownum=1; commit; insert into tv values(seqt1.nextval,200000*dbms_random.value,'MACLEAN IS UPDATING'); commit; insert into tv values(seqt1.nextval,300000*dbms_random.value,'MACLEAN IS UPDATING'); commit; update tv set t2=t2+10 where rownum=1; commit; dbms_lock.sleep(1); end loop;end;//* 在源端打开模拟的应用程序 *//* 在target插入一条额外的纪录 */SQL> insert into tv values(99993242,1213,'OUT OF SYNC');1 row created.SQL> commit;Commit complete.再次运行Goldengate veridata中相关的job，可以在job相关的日志中看到详细的out-of-sync纪录信息:
Profile: newpGeneral:*********************************************************************** Out-Of-Sync Output Format: binary Maximum Size of Each Out-Of-Sync XML Chunk (Rows): 500 Output in-sync rows to OOS file: falseOutput in-sync after in-flight rows to OOS file: true Report in-sync rows to report file: falseReport in-sync after in-flight rows to report file: false Sorting Method: Sort Data Using: database Maximum Memory Usage (MB): 50 Number Of Concurrent Threads: 4 Temporary Storage Directory for Source Data: Temporary Storage Directory for Target Data: NSort Memory Retry Limit: 0NSort Memory Retry Wait Interval: 0Initial Compare (General):*********************************************************************** Max Concurrent Comparison Threads: 4Terminate when Maximum Records Out-Of-Sync: 100000 Output Out-Of-Sync Record Details to Report File: false Update Report file Every (seconds): 0 Update Report file Every Threads: 0Terminate when Maximum Records Out-Of-Sync: 100000 Output Out-Of-Sync Record Details to Report File: false Update Report file Every (seconds): 0 Update Report file Every (records): 0 Limit Number of Input Rows: 0 Delta processsing enabled: falseInitial Compare (Event Reporting):*********************************************************************** Generate Messages: both Generate Warning Messages For Out-Of-Sync Rows After (differences): 9999Initial Compare (Agent):*********************************************************************** Use Static Listening Port For For Agent During Row Hash On Source: 0 Use Static Listening Port For For Agent During Row Hash On Target: 0Initial Compare (NonStop Process):*********************************************************************** Source Process Name Starting With: Source Process CPU Number: -1 Source Process Priority: 0 Target Process Name Starting With: Target Process CPU Number: -1 Target Process Priority: 0Confirm-Out-Of-Sync (General):*********************************************************************** Perform Confirm Out-Of-Sync Step: true Run Concurrently With Initial Compare: true Delay Confirm-Out-Of-Sync By (seconds): 0Terminate when Maximum Records Out-Of-Sync: 100000 Output Out-Of-Sync Record Details to Report File: true Update Report file Every (seconds): 0 Update Report file Every (records): 0Confirm-Out-Of-Sync (Event Reporting):*********************************************************************** Generate Messages: both Generate Warning Messages For For Out-Of-Sync Rows After (differences): 9999Confirm-Out-Of-Sync (Agent):*********************************************************************** Use Static Listening Port For Agent During Row Hash On Source: 0 Use Static Listening Port For Agent During Row Hash On Target: 0Confirm-Out-Of-Sync (NonStop Process):*********************************************************************** Source Process Name Starting With: Source Process CPU Number: -1 Source Process Priority: 0 Target Process Name Starting With: Target Process CPU Number: -1 Target Process Priority: 0Starting Veriagt session for source rowhash at 2010-12-11 20:32:25.Initial Compare Source Agent Information: ORACLE_HOME : /s01/10gdb NLS_LANG (from environment) : ORA_SDTZ : NLS_LANG (from DBMS) : AMERICAN_AMERICA.WE8ISO8859P1 NLS_LENGTH_SEMANTICS : BYTE SESSIONTIMEZONE : -05:00 Process ID : 21473 Operating System : Linux Architecture : x86_64 Report File : /home/maclean/veridata/agent/dirver/report/maclean_PC_4150_000010110000000100000001___source_rh.rpt Trace File : /home/maclean/veridata/agent/dirver/trace/maclean_PC_4150_000010110000000100000001___source_rh.trcStarting Veriagt session for target rowhash at 2010-12-11 20:32:26.Initial Compare Target Agent Information: ORACLE_HOME : /s01/db_1 NLS_LANG (from environment) : ORA_SDTZ : NLS_LANG (from DBMS) : AMERICAN_AMERICA.WE8ISO8859P1 NLS_LENGTH_SEMANTICS : BYTE SESSIONTIMEZONE : +08:00 Process ID : 18639 Operating System : Linux Architecture : x86_64 Report File : /home/maclean/veridata/agent/dirver/report/maclean_PC_4150_000010110000000100000001___target_rh.rpt Trace File : /home/maclean/veridata/agent/dirver/trace/maclean_PC_4150_000010110000000100000001___target_rh.trcProcessing first rowhash block from source at 2010-12-11 20:32:26.Processing first rowhash block from target at 2010-12-11 20:32:27.Comparing first row at 2010-12-11 20:32:27.Starting Veriagt session for source COOS at 2010-12-11 20:32:27.Performance Statistics for source Rowhash at 2010-12-11 20:32:27. rows: 103959 duration (secs): 00:00:02 rows/sec: 51979.50 row bytes: 4345147 row bytes/sec: 2172573 bytes/row: 41 rh bytes/row: 23 rows skipped: 0 blocks skipped: 0 hash comp rate: 0.55 total comp rate: 0.23 pct time fetching: 15.44 pct time waiting: 5.91 time until first row: 00:00:00 ipc msgs: 12 ipc bytes: 2809974 bytes/msg: 234164 compressed bytes/msg: 99262 bytes/sec: 1404987 compressed bytes/sec: 595572 msg compression ratio: 0.42*** Summarizing Initial Row Comparison Step *** Time: 2010-12-11 20:32:27. Elapsed Time: 00:00:00 Comparisons performed: 103961 Rows per second: 169593 Rows out-of-sync: 11 inserts: 9 updates: 0 deletes: 2Performance Statistics for target Rowhash at 2010-12-11 20:32:27. rows: 103952 duration (secs): 00:00:01 rows/sec: 103952.00 row bytes: 4344826 row bytes/sec: 4344826 bytes/row: 41 rh bytes/row: 23 rows skipped: 0 blocks skipped: 0 hash comp rate: 0.55 total comp rate: 0.23 pct time fetching: 20.51 pct time waiting: 17.15 time until first row: 00:00:00 ipc msgs: 12 ipc bytes: 2809779 bytes/msg: 234148 compressed bytes/msg: 99256 bytes/sec: 2809779 compressed bytes/sec: 1191076 msg compression ratio: 0.42COOS source Agent Information: ORACLE_HOME : /s01/10gdb NLS_LANG (from environment) : ORA_SDTZ : NLS_LANG (from DBMS) : AMERICAN_AMERICA.WE8ISO8859P1 NLS_LENGTH_SEMANTICS : BYTE SESSIONTIMEZONE : -05:00 Process ID : 21475 Operating System : Linux Architecture : x86_64 Report File : /home/maclean/veridata/agent/dirver/report/maclean_PC_4150_000010110000000100000001___source_coos.rpt Trace File : /home/maclean/veridata/agent/dirver/trace/maclean_PC_4150_000010110000000100000001___source_coos.trcStarting Veriagt session for target COOS at 2010-12-11 20:32:28.COOS target Agent Information: ORACLE_HOME : /s01/db_1 NLS_LANG (from environment) : ORA_SDTZ : NLS_LANG (from DBMS) : AMERICAN_AMERICA.WE8ISO8859P1 NLS_LENGTH_SEMANTICS : BYTE SESSIONTIMEZONE : +08:00 Process ID : 18641 Operating System : Linux Architecture : x86_64 Report File : /home/maclean/veridata/agent/dirver/report/maclean_PC_4150_000010110000000100000001___target_coos.rpt Trace File : /home/maclean/veridata/agent/dirver/trace/maclean_PC_4150_000010110000000100000001___target_coos.trcPersistent out of sync row (optype: insert) source row values: T1 K 0 6 107913 GGROWHASH 1 8 0x0E8563AC22AA9C0F T2 H 2 5 55512 T3 H 3 19 MACLEAN IS UPDATINGPersistent out of sync row (optype: insert) source row values: T1 K 0 6 107914 GGROWHASH 1 8 0xAAB002BF5C1B3233 T2 H 2 6 255834 T3 H 3 19 MACLEAN IS UPDATINGPersistent out of sync row (optype: insert) source row values: T1 K 0 6 107917 GGROWHASH 1 8 0xC0723BE324DC032D T2 H 2 6 197710 T3 H 3 19 MACLEAN IS UPDATINGPersistent out of sync row (optype: insert) source row values: T1 K 0 6 107918 GGROWHASH 1 8 0x8A63DA5F06C3A12C T2 H 2 6 194628 T3 H 3 19 MACLEAN IS UPDATINGPersistent out of sync row (optype: delete) target row values: T1 K 0 8 99993242 GGROWHASH 1 8 0x958B6304CEC4BAD3 T2 H 2 4 1213 T3 H 3 11 OUT OF SYNC*** Summarizing Confirm Out-of-Sync Step *** Time: 2010-12-11 20:32:29. Elapsed Time: 00:00:01 Rows confirmed: 11 Rows per second: 11*** Summarizing Persistently Out-of-Sync Rows *** Rows out-of-sync: 5 inserts: 4 updates: 0 deletes: 1*** Summarizing In-Sync-after-In-Flight Rows *** Rows in-sync-after-in-flight: 6 inserts: 5 updates: 0 deletes: 1*** Summarizing Still-Changing Rows *** Rows in-flight: 0 inserts: 0 updates: 0 deletes: 0Comparison terminated normally.