postgresql slow: primary key on 80M row table -

-

i've 64gb ram machine, 20 cpus, , command create primary key on table 80m rows:

alter table wikipedia_article add constraint pagelinks_pkey primary key (language, title);

the problem i've got that:

  • only 1 cpu used 100% (avg load 99%), rest aren't used @ all
  • the write speed during primary key creation quite low, 2.6 m/s, read speed missing report produced pg_activity

this table structure:

                          table "public.wikipedia_article"     column    |       type       | modifiers | storage  | stats target | description --------------+------------------+-----------+----------+--------------+-------------  language     | text             | not null  | extended |              |  title        | text             | not null  | extended |              |  langcount    | integer          |           | plain    |              |  othercount   | integer          |           | plain    |              |  totalcount   | integer          |           | plain    |              |  lat          | double precision |           | plain    |              |  lon          | double precision |           | plain    |              |  importance   | double precision |           | plain    |              |  osm_type     | character(1)     |           | extended |              |  osm_id       | bigint           |           | plain    |              |  infobox_type | text             |           | extended |              |  population   | bigint           |           | plain    |              |  website      | text             |           | extended |              | has oids: no

the import happens automatically via pg_restore, source

http://www.nominatim.org/data/wikipedia_article.sql.bin

any ideas on can try make things faster? changed value of conf varible "maintenance_work_mem" half of ram size. tried change of kernel settings, no joy:

sysctl -w kernel.shmmax=17179869184 sysctl -w kernel.shmall=1048576 sysctl vm.overcommit_memory=1  sysctl vm.swappiness=10

the os running on vm, in digital ocean, on ssd drives, expecting work faster such vm configuration.

server info: postgresql 9.3.13 on x86_64-unknown-linux-gnu, compiled gcc (ubuntu 4.8.4-2ubuntu1~14.04.1) 4.8.4, 64-bit

many thanks, vg

postgresql uses 1 statement 1 cpu - cpu utilization 100% expected. can try set maintenance_work_mem higher value. default low - can try '2gb' example.


Comments

Post a Comment

Popular posts from this blog

Spring Boot + JPA + Hibernate: Unable to locate persister -

-
i trying work entity inside request method of spring boot application. entity in project , pom.xml has dependencies it. why error message?: org.hibernate.unknownentitytypeexception: unable locate persister: com.ric.bill.model.bs.par package com.ric.web; import java.util.concurrent.atomic.atomiclong; import javax.persistence.entitymanager; import javax.persistence.persistencecontext; import net.sf.ehcache.cachemanager; import net.sf.ehcache.management.cache; import net.sf.ehcache.management.cachestatistics; import org.hibernate.stat.statistics; import org.springframework.beans.factory.annotation.autowired; import org.springframework.boot.autoconfigure.domain.entityscan; import org.springframework.cache.annotation.enablecaching; import org.springframework.transaction.annotation.enabletransactionmanagement; import org.springframework.transaction.annotation.propagation; import org.springframework.transaction.annotation.transactional; import org.springframework.web.bind.annotati
Read more

go - Golang: panic: runtime error: invalid memory address or nil pointer dereference using bufio.Scanner -

-
i implementing go program uses bufio.scanner , bufio.writer have packaged code follows package main import ( "fmt" "player/command" "strings" ) func main() { //enter code here. read input stdin. print output stdout commands.scanner.scan() { //scan new line , send comand variable check command exist or not input := strings.split(strings.trim(commands.scanner.text(), " "), " ") command := input[0] if command == "" { fmt.printf("$ %s:", commands.pwd) continue } if !commands.commands[command] { commands.throwerror("cannot recognize input.") } else { commands.execute(command, input[1:], nil) } fmt.printf("$ %s:", commands.pwd) } } i using init.go file in main package follows package main import ( "flag" "player/sourc
Read more

c - double free or corruption (fasttop) -

-
this code reads null terminated string file (file containing null terminated strings & should provide index wanted string). error: *** error in `./main': double free or corruption (fasttop): 0x090a4a80 *** aborted code: char *tmp_realloc=null; char *sstring=null; int i=1; /*set file pointer point string*/ fseek(fh,index,seek_set); sstring=(char*)malloc(sizeof(char)); if(sstring == null) return null; tmp_realloc=sstring; while( (*(sstring+i-1)=(char)fgetc((file*)fh)) != 0x0 ) { /*reallocate more memory*/ tmp_realloc=(char*)realloc((char*)sstring,++i); /*if not same address copy old new & set old point new*/ if(tmp_realloc != sstring){ strcpy((char*)tmp_realloc,(char*)sstring); free(sstring); sstring=tmp_realloc; } } could tell me pointer causing issue (& reason) ? remove part /*if not same address copy old new & set old point new*/ if(tmp_realloc != sstring){ strc
Read more