In a predetermined format, csv files can be uploaded and in the process it gets pursed through an algorithm every time we upload a dataset. One of the categories for check is the name of the countries in the world and country groups (continents and/or regions) to categorize the data. Recently, we are trying to add international organization names to this list as the data can be categorized based on this. To implement the list of countries and country groups, Ruby script that reads data from a CSV file (data/wesis_unique_cow_entities_v2020-06-23.csv), processes it, and imports it into a database table.
'create_country' method creates/updates a Country record in the database based on the data provided in the current hash. All the country group relations are provided through a json file. After adding the international organization names and codes in the country and country groups, I ran the code in localhost. It performed the DB migration on its own and the code is working perfectly. I wanted to test it in the staging server. The deployment runs without any errors and the branch is active in the server. But the changes regarding international organizations are not visible. I checked the staging server DB (postgreSQL) and the entries are not there. We have erased the cache and redeployed the branch, sill the same. Can anyone give me an idea what can we do to resolve this?
I have checked the table contents in the database and the new country name and groups entities are not available in the database. I have erased the DB cache in case the previous entries are stored there. Instead of a new file name with updated data, I kept the old filename but changed the file content. Tried erasing one of the existing country group relations to see if the changes are really being implemented, it could not find the erased entry so it is working as expected.
My expectation from the changes is that the new entries would get updated the staging DB and subsequently, in the production server. Also, the user is able to choose the international organizations in the drop down in the data explorer page to look for relevant data.
require 'rubygems'
require 'roo'
file = SmarterCSV.process("data/wesis_unique_cow_entities_v2020-06-23.csv",{ remove_empty_values: true, col_sep: ';'})
def create_country current
country = Country.where( "entity_id = ? ", current[:cow_code].to_i).take
if country.nil?
country = Country.new
end
country.entity_id = current[:cow_code].to_i
country.entity_name = current[:country_name]
unless current[:independent_list].nil?
independence_list = relation_year_string_to_array current[:independent_list]
independence_hash = Hash.new
independence_list.each do |tuple|
independence_hash[tuple[0].to_i] = tuple[1].to_i
end
country.independence_hash = independence_hash
end
if current[:pop_over_500k].kind_of? Integer
country.pop_over_500k = current[:pop_over_500k] == 1
elsif current[:pop_over_500k].kind_of? String
country.pop_over_500k = current[:pop_over_500k].delete(" ").eql?("1")
end
country.wiki_url = current[:wiki_url]
unless current[:alternative_names].nil?
alternative_names = current[:alternative_names].split(",").map!{|s| s.strip}
country.alternative_names = alternative_names
end
unless current[:remark].nil?
remark = current[:remark].split(";").map!{|s| s.strip}
country.remark = remark
end
unless current[:vdem_historical_names].nil?
vdem_historical_names = current[:vdem_historical_names].split(";").map!{|s| s.strip}
country.vdem_historical_names = vdem_historical_names
end
unless current[:cow_abb].nil?
cow_abb = current[:cow_abb].strip
country.cow_abb = cow_abb
end
unless current[:iso3c].nil?
iso3c_abb = current[:iso3c].strip
country.iso3c_abb = iso3c_abb
end
unless current[:iso3n].nil?
iso3n_abb = current[:iso3n]
country.iso3n_abb = iso3n_abb.to_i
end
country.save!
if(!current[:related_list].nil? && !current[:relation_year].nil? && !current[:relation_type].nil?)
create_relations current, country
end
end
def string_to_array string_array
#trim the brackets and delete white space
trimmed_string = string_array[1..-2].delete(" ")
return trimmed_string.split(",")
end
def relation_year_string_to_array string_array
#trim the brackets and delete white space
trimmed_string = string_array[1..-2].delete(" ")
full_array = []
tuple_item1 = ""
tuple_item2 = ""
started_first = false
started_second = false
trimmed_string.split('').each do |character|
if character.eql?("(")
started_first = true
next
elsif character.eql?(",")
if(started_first == true)
started_second = true
end
started_first = false;
next
elsif character.eql?(")")
started_second = false
full_array.push([tuple_item1, tuple_item2])
tuple_item1 = ""
tuple_item2 = ""
end
if started_first == true
tuple_item1 += character
elsif started_second == true
tuple_item2 += character
end
end
return full_array
end
def create_relations current, country
related_list = string_to_array current[:related_list]
relation_year_list = relation_year_string_to_array current[:relation_year]
relation_type_list = string_to_array current[:relation_type]
if(related_list.count > relation_type_list.count || related_list.count > relation_year_list.count)
#if there are more relation ids than relation types or relation years
return
end
related_list.each_with_index do |related_id, index|
relation_start = relation_year_list[index][0].to_i
relation_end = relation_year_list[index][1].to_i
relation_type = relation_type_list[index].to_i
unless country.nil?
faulty_rel = Relation.where("country_id =? AND (relation_start =? OR relation_end =? OR relation_type =? )", country, 0, 0, 0).take
if(!faulty_rel.nil?)
faulty_rel.destroy!
end
check = Relation.where("country_id = ? AND related_country_cow_code = ? AND relation_start = ? AND relation_type = ?",
country, related_id.to_i, relation_start, relation_type).take
if !check.nil?
if check.update!(country_id: country.id,
related_country_cow_code: related_id.to_i,
relation_start: relation_start,
relation_end: relation_end,
relation_type: relation_type)
end
next
end
relation = Relation.new
relation.country = country
relation.related_country_cow_code = related_id.to_i
relation.relation_start = relation_start
relation.relation_end = relation_end
relation.relation_type = relation_type
relation.save!
end
end
end
file.each_with_index do |current, index|
create_country current
end
[enter image description here](https://i.sstatic.net/2Cmx8.jpg)
