Problems with german characters (de_DE.utf8) using trac importer

[thiema]

Hello,

I am trying to import a companies trac project into jira. Currently I have a problem when importing issues containing special german chars. It seems that all umlauts occurring in the trac ticket description are not correctly recognized. After the import they show up as questionmarks (?).

I made sure that all 'layers' and components are configured to us utf8:

• the trac db,
• the os running trac
• jira-app
• jira-db (postgres)
• both os running the db and the app

Actually it somehow works: Umlauts used in the trac ticket summary are correctly handled during import. No problem here. Also when creating new issues in jira: no problem here ! Importing issues with umlauts using csv importer works as well. No problems with umlauts here !

I checked the postgres db: The questionmarks occur already in the jiraissue table (column description). I noticed that the type postgres db type is VARCHAR rsp. TEXT for column summary rsp. description. However, I dont believe that this matters.

Any ideas ?

Internationalization settings:

Indexing language	English - Aggressive Stemming
Installed languages	English (UK) English (United States) French (France) German (Germany) Japanese (Japan) Spanish (Spain)
Default language	English (United States)
Default user time zone	(GMT-06:00) GMT-6

some more settings:

Java Version	1.7.0
Java Vendor	Oracle Corporation
JVM Version	1.7
JVM Vendor	Oracle Corporation
JVM Implementation Version	21.0-b17
Java Runtime	Java(TM) SE Runtime Environment
Java VM	Java HotSpot(TM) 64-Bit Server VM
User Name	jira
User Timezone	Europe/Berlin
User Locale	English (United States)
System Encoding	ANSI_X3.4-1968
Operating System	Linux 3.13.0-77-generic
OS Architecture	amd64
Application Server Container	Apache Tomcat/7.0.55
Database type	postgres72
Database JNDI address	postgres72 jdbc:postgresql://172.17.0.3:5432/postgres
Database URL	jdbc:postgresql://172.17.0.3:5432/postgres
Database version	9.4.0
Database driver	PostgreSQL Native Driver PostgreSQL 9.0 JDBC4 (build 801)

JIRA info:

Version	6.3.15
Build Number	6346
Build Date	Tue Feb 03 00:00:00 CET 2015
Build Revision	dbc023dd75cecacf443c4b235f66124b15f5c5fe

[Mohamad Khalife]

The cause of this problem is the difference between the encoding in the Excel file and CSV. The following steps shows how to solve it:

1. Have a clean export Excel file ready to be imported
2. Make sure the file doesn't contain any commas
3. Save the file as Unicode Text (*.txt)
4. Now you will have a text file with data separated by tabs
5. Open the file in Notepad
6. Select the space between two different values and copy it
7. Click ctrl+H to open the replace toolbox
8. Paste what you have selected in the Find what text field
9. Type "," in the Replace with text field
10. Click Replace All
11. Copy all data from that file.
12. Create a new text file and paste the copied data.
13. Save the text file with Unicode encoding.
14. Change its extension to CSV.
15. Now if you open the saved CSV file, you will see that it is not delimited based on ",". To solve it:
    1. From the same opened CSV file, click on "Data"
    2. Click on "From Text"and Browse for the new saved CSV file and click on "Open"
    3. Choose "Delimited", Next"
    4. Uncheck all values except for "Comma" and make sure that "Treat consecutive delimiters as one" is unchecked
    5. Click on "Next", "Finish" and "Ok"
16. Now you will have your CSV file ready for import

[thiema]

Hello Moe,

I should have mentioned, that I am using the "Trac Importer" (https://confluence.atlassian.com/jira/importing-data-from-trac-238617182.html). Hence there is no csv or excel data. I directly import the zipped trac.db and other files, as described in the reference.

Regarding csv imports in general: I tried to import a test csv file and found there is no problem with character encoding. However, currently I think using csv import method is no good option when importing the entire project from trac with thousands of issues.