pgloader:堆耗尽,游戏结束。问题

编写代码后,当我运行pgloader脚本时,它将以这种方式失败:

load database
    from mysql://xxx:yyy127.0.0.1/zzz
    into pgsql://xxx:yyy@localhost/zzz

 including only table names matching 'TABLE'

with
data only,create no tables,preserve index names,batch rows = 1000,batch size = 500 MB,prefetch rows = 1000
-- on error stop,set work_mem to '2048 MB',maintenance_work_mem to '4096 MB';



-- before load do $$ drop schema if exists jobs cascade; $$;

我收到此错误消息,但这是间歇性的,也就是说,它并不总是发生,并且我不确定要输入什么参数。我有很多公羊,每条记录大约50kb。

2020-05-25T04:28:14.194000Z INFO Incomplete Foreign Key definition: constraint "fk_job_currency1" on table "jobs2_beta2.job" referencing table NIL
2020-05-25T04:28:14.194000Z INFO Incomplete Foreign Key definition: constraint "fk_job_job_category1" on table "jobs2_beta2.job" referencing table NIL
2020-05-25T04:28:14.194000Z INFO Incomplete Foreign Key definition: constraint "fk_job_organization1" on table "jobs2_beta2.job" referencing table NIL
2020-05-25T04:28:14.194000Z INFO Incomplete Foreign Key definition: constraint "fk_job_resource1" on table "jobs2_beta2.job" referencing table NIL
2020-05-25T04:28:14.194000Z INFO Incomplete Foreign Key definition: constraint "fk_job_user1" on table "jobs2_beta2.job" referencing table NIL
2020-05-25T04:28:14.198000Z SQL MySQL: sending query: -- params: db-name
--         table-type-name
--         only-tables
--         only-tables
--         including
--         filter-list-to-where-clause incuding
--         excluding
--         filter-list-to-where-clause excluding
  SELECT table_name,index_name,index_type,sum(non_unique),cast(GROUP_concat(column_name order by seq_in_index) as char)
    FROM information_schema.statistics
   WHERE table_schema = 'jobs2_beta2'
         and (table_name = 'job')

GROUP BY table_name,index_type;
2020-05-25T04:28:14.225000Z INFO Processing source catalogs
2020-05-25T04:28:14.272000Z NOTICE Prepare PostgreSQL database.
2020-05-25T04:28:14.275000Z DEBUG CONNECTED TO #<PGLOADER.PGSQL:PGSQL-CONNECTION pgsql://ziprecruiter@localhost:5432/ziprecruiter {1006A2A683}>
2020-05-25T04:28:14.275000Z DEBUG SET client_encoding TO 'utf8'
2020-05-25T04:28:14.275000Z DEBUG SET work_mem TO '1024 MB'
2020-05-25T04:28:14.276000Z DEBUG SET maintenance_work_mem TO '4096 MB'
2020-05-25T04:28:14.276000Z DEBUG SET application_name TO 'pgloader'
2020-05-25T04:28:14.280000Z DEBUG BEGIN
2020-05-25T04:28:14.314000Z SQL DROP TABLE IF EXISTS jobs2_beta2.job CASCADE;
2020-05-25T04:28:15.316000Z SQL CREATE TABLE jobs2_beta2.job
(
  id                       bigserial not null,resource_id              bigint not null,url                      text not null,job_title                text,html_job_description     text,text_job_description     text,last_crawl_date          timestamptz,first_indexed_date       timestamptz,job_category_id          bigint not null,currency_id              bigint not null,salary_exact             double precision,salary_is_range          smallint,salary_range_start       double precision,salary_range_end         double precision,address                  text,source_program           varchar(255),organization             varchar(255),organization_count       bigint,expiration_date          timestamptz,is_sponsored             smallint,is_hidden                smallint,educational_requirements text,experience_requirements  text,destination              text,organization_id          bigint not null,user_id                  bigint not null,is_expired               smallint,salary_periodicity       varchar(255),json_schema              text,clean_job_description    text,estimated_job_category   varchar(255)
);
2020-05-25T04:28:15.326000Z SQL -- params: table-names
select n,n::regclass::oid
  from (values ('jobs2_beta2.job')) as t(n);

2020-05-25T04:28:15.392000Z NOTICE COPY jobs2_beta2.job
2020-05-25T04:28:15.392000Z DEBUG Reader started for jobs2_beta2.job
2020-05-25T04:28:15.405000Z DEBUG start jobs2_beta2.job   1400
2020-05-25T04:28:15.407000Z INFO COPY ON ERROR STOP
2020-05-25T04:28:15.408000Z DEBUG CONNECTED TO #<MYSQL-CONNECTION mysql://jobs_client2@127.0.0.1:3306/jobs2_beta2 {100A4E61B3}>
2020-05-25T04:28:15.408000Z SQL MySQL: sending query: SELECT `id`,`resource_id`,`url`,`job_title`,`html_job_description`,`text_job_description`,`last_crawl_date`,`first_indexed_date`,`job_category_id`,`currency_id`,`salary_exact`,`salary_is_range`,`salary_range_start`,`salary_range_end`,`address`,`source_program`,`organization`,`organization_count`,`expiration_date`,`is_sponsored`,`is_hidden`,`educational_requirements`,`experience_requirements`,`destination`,`organization_id`,`user_id`,`is_expired`,`salary_periodicity`,`json_schema`,`clean_job_description`,`estimated_job_category` FROM `job`
2020-05-25T04:28:15.416000Z DEBUG CONNECTED TO #<PGLOADER.PGSQL:PGSQL-CONNECTION pgsql://ziprecruiter@localhost:5432/ziprecruiter {100A8C11A3}>
2020-05-25T04:28:15.416000Z DEBUG SET client_encoding TO 'utf8'
2020-05-25T04:28:15.416000Z DEBUG SET work_mem TO '1024 MB'
2020-05-25T04:28:15.416000Z DEBUG SET maintenance_work_mem TO '4096 MB'
2020-05-25T04:28:15.416000Z DEBUG SET application_name TO 'pgloader'
2020-05-25T04:28:15.416000Z SQL SET search_path TO jobs2_beta2;
2020-05-25T04:28:15.417000Z INFO pgsql:copy-rows-from-queue[0]: jobs2_beta2.job (id resource_id url job_title
                                                html_job_description
                                                text_job_description
                                                last_crawl_date
                                                first_indexed_date
                                                job_category_id currency_id
                                                salary_exact salary_is_range
                                                salary_range_start
                                                salary_range_end address
                                                source_program organization
                                                organization_count
                                                expiration_date is_sponsored
                                                is_hidden
                                                educational_requirements
                                                experience_requirements
                                                destination organization_id
                                                user_id is_expired
                                                salary_periodicity json_schema
                                                clean_job_description
                                                estimated_job_category)

Gen  Boxed Unboxed   LgBox LgUnbox  Pin       Alloc     Waste        Trig      WP GCs Mem-age
 0       0       0       0       0    0           0         0    42949672       0   0  0.0000
 1     577   42719       0      36    5  1301487024 118415952   753961944       0   1  1.3187
 2    1776   83008      25      60   65  2579123472 201863920     2000000     668   0  0.8672
 3       0       0       0       0    0           0         0     2000000       0   0  0.0000
 4       0       0       0       0    0           0         0     2000000       0   0  0.0000
 5       0       0       0       0    0           0         0     2000000       0   0  0.0000
 6    1593    1278       0       0    0    90993120   3083808     2000000    1501   0  0.0000
 7       0       0       0       0    0           0         0     2000000       0   0  0.0000
           Total bytes allocated    =    3971603616
           Dynamic-space-size bytes =    4294967296
GC control variables:
   *GC-INHIBIT* = true
   *GC-PENDING* = true
   *STOP-FOR-GC-PENDING* = false
fatal error encountered in SBCL pid 9514(tid 0x7ffff492f700):
Heap exhausted,game over.

Welcome to LDB,a low-level debugger for the Lisp runtime environment.
ldb>


我尝试摆弄参数,有人可以在这个问题上提供帮助吗?

iCMS 回答:pgloader:堆耗尽,游戏结束。问题

发现与“堆耗尽,游戏结束”有关的github问题,可能会有所帮助。

https://github.com/dimitri/pgloader/issues/327

本文链接:https://www.f2er.com/2280379.html

大家都在问