Showing posts with label csv. Show all posts
Showing posts with label csv. Show all posts

Monday, 17 July 2017

How to export data from SQL Server 2005 to MySQL

Question: I've been banging my head against SQL Server 2005 trying to get a lot of data out. I've been given a database with nearly 300 tables in it and I need to turn this into a MySQL database. My first call was to use bcp but unfortunately it doesn't produce valid CSV - strings aren't encapsulated, so you can't deal with any row that has a string with a comma in it (or whatever you use as a delimiter) and I would still have to hand write all of the create table statements, as obviously CSV doesn't tell you anything about the data types.
What would be better is if there was some tool that could connect to both SQL Server and MySQL, then do a copy. You lose views, stored procedures, trigger, etc, but it isn't hard to copy a table that only uses base types from one DB to another... is it?
Does anybody know of such a tool? I don't mind how many assumptions it makes or what simplifications occur, as long as it supports integer, float, datetime and string. I have to do a lot of pruning, normalising, etc. anyway so I don't care about keeping keys, relationships or anything like that, but I need the initial set of data in fast!

Solution: Using MSSQL Management Studio i've transitioned tables with the MySQL OLE DB. Right click on your database and go to "Tasks->Export Data" from there you can specify a MsSQL OLE DB source, the MySQL OLE DB source and create the column mappings between the two data sources.
You'll most likely want to setup the database and tables in advance on the MySQL destination (the export will want to create the tables automatically, but this often results in failure). You can quickly create the tables in MySQL using the "Tasks->Generate Scripts" by right clicking on the database. Once your creation scripts are generated you'll need to step through and search/replace for keywords and types that exist in MSSQL to MYSQL.
Of course you could also backup the database like normal and find a utility which will restore the MSSQL backup on MYSQL. I'm not sure if one exists however.

Sunday, 16 July 2017

Java lib or app to convert CSV to XML file?

Question: Is there an existing application or library in Java which will allow me to convert a CSV data file to XML file?
The XML tags would be provided through possibly the first row containing column headings

Solution:  OpenCsv for parsing CSV (small, simple, reliable and easy to use)
Xstream to parse/serialize XML (very very easy to use, and creating fully human readable xml) Using the same sample data as above, code would look like:

import java.io.FileReader;
import java.io.FileWriter;
import java.util.ArrayList;
import java.util.List;

import au.com.bytecode.opencsv.CSVReader;

import com.thoughtworks.xstream.XStream;

public class CsvToXml {     

    public static void main(String[] args) {

        String startFile = "./startData.csv";
        String outFile = "./outData.xml";

        try {
            CSVReader reader = new CSVReader(new FileReader(startFile));
            String[] line = null;

            String[] header = reader.readNext();

            List out = new ArrayList();

            while((line = reader.readNext())!=null){
                List<String[]> item = new ArrayList<String[]>();
                    for (int i = 0; i < header.length; i++) {
                    String[] keyVal = new String[2];
                    String string = header[i];
                    String val = line[i];
                    keyVal[0] = string;
                    keyVal[1] = val;
                    item.add(keyVal);
                }
                out.add(item);
            }

            XStream xstream = new XStream();

            xstream.toXML(out, new FileWriter(outFile,false));

        } catch (Exception e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }
}

Producing the following result: (Xstream allows very fine tuning of the result...)


<list>
  <list>
    <string-array>
      <string>string</string>
      <string>hello world</string>
    </string-array>
    <string-array>
      <string>float1</string>
      <string>1.0</string>
    </string-array>
    <string-array>
      <string>float2</string>
      <string>3.3</string>
    </string-array>
    <string-array>
      <string>integer</string>
      <string>4</string>
    </string-array>
  </list>
  <list>
    <string-array>
      <string>string</string>
      <string>goodbye world</string>
    </string-array>
    <string-array>
      <string>float1</string>
      <string>1e9</string>
    </string-array>
    <string-array>
      <string>float2</string>
      <string>-3.3</string>
    </string-array>
    <string-array>
      <string>integer</string>
      <string>45</string>
    </string-array>
  </list>
  <list>
    <string-array>
      <string>string</string>
      <string>hello again</string>
    </string-array>
    <string-array>
      <string>float1</string>
      <string>-1</string>
    </string-array>
    <string-array>
      <string>float2</string>
      <string>23.33</string>
    </string-array>
    <string-array>
      <string>integer</string>
      <string>456</string>
    </string-array>
  </list>
  <list>
    <string-array>
      <string>string</string>
      <string>hello world 3</string>
    </string-array>
    <string-array>
      <string>float1</string>
      <string>1.40</string>
    </string-array>
    <string-array>
      <string>float2</string>
      <string>34.83</string>
    </string-array>
    <string-array>
      <string>integer</string>
      <string>4999</string>
    </string-array>
  </list>
  <list>
    <string-array>
      <string>string</string>
      <string>hello 2 world</string>
    </string-array>
    <string-array>
      <string>float1</string>
      <string>9981.05</string>
    </string-array>
    <string-array>
      <string>float2</string>
      <string>43.33</string>
    </string-array>
    <string-array>
      <string>integer</string>
      <string>444</string>
    </string-array>
  </list>
</list>