Reading from a DB2 CLOB into a String with Groovy 1.56

Groovy makes everything easier, so I expected that there would be a simple, Groovy way to read text out of a CLOB field in my DB2 database. So far, however, I haven’t found any reference to one.

The java.sql.Clob API Guide describes three different methods that can be used to read from a CLOB: getAsciiStream(), getCharacterStream() and getSubString(). I worked out a few different uses from Sun’s suggestions:

Using getAsciiStream():

sql.eachRow("select clob_field from table_x") {row ->
java.sql.Clob clob = (java.sql.Clob) row["clob_field"]
String valueFromClob = clob.getAsciiStream().getText()
}

Using getSubString():

sql.eachRow("select clob_field from table_x") {row ->
java.sql.Clob clob = (java.sql.Clob) row["clob_field"]
String valueFromClob = clob.getSubString(1, clob.length().intValue())
}

Using getCharacterStream():

sql.eachRow("select clob_field from table_x") {row ->
java.sql.Clob clob = (java.sql.Clob) row["clob_field"]
List lines = clob.getCharacterStream().readLines()
StringBuffer contents = new StringBuffer()
lines.each {
contents.append(it)
}
String valueFromClob = contents.toString()
}

I worked those into a handy utility class:


import java.sql.Clob
import org.apache.log4j.Logger

public class ClobUtil {

static String getAsString(Object rsField) {
Logger log = Logger.getLogger("ClobUtil")
int tooBig = (64 * 1024 * 1024)

Clob clob = (Clob) rsField
/*
* In practice, we have found that the JVM has a 64K limit on Strings and will
* throw a java.lang.ClassFormatError when trying to process a longer String
* value. See http://jira.codehaus.org/browse/GROOVY-2382.
*/
int contentSize = clob.length().intValue()
if (clob.length() > tooBig) {
log.warn("Truncating contents at 64K characters. Use getAsCharArray() to read larger values.")
contentSize = tooBig
}

return clob.getSubString(1, contentSize)
}

static char[] getAsCharArray(Object rsField) {
Clob clob = (Clob) rsField
Reader reader = clob.getCharacterStream()
List contents = []
for (i in 1..clob.length().intValue()) {
contents << reader.read()
}
return (char[]) contents.toArray()
}
}

#clob, #db2, #groovy, #groovy-sql

Importing projects from IntelliJ IDEA to NetBeans

Googling for how to import an IntelliJ project into NetBeans didn’t turn up much information. NetBeans has a nice import tool form migrating from Eclipse — clearly, Sun knows who their competition is — but there’s nothing like that for other IDEs. And that makes sense: who would go back to a free IDE after shelling out for IntelliJ licenses? (Oh, the irony.)

new-project

This was kind of counter-intuitive to me, but what I learned is that it’s probably best to import a Groovy or Grails project as a Java Free-Form Project. The Free-Form Project lets you retain control of your build script (build.xml), rather than letting NetBeans alter it for you. It also lets you specify a number of project properties while you’re setting it up, including preferred JDK and classpath.

There’s also a plugin you can install, called Automatic Projects. With Automatic Projects installed, you can open any existing project, and NetBeans will simply scan the directory and configure project settings for itself.

I’m running into a problem. NetBeans doesn’t seem to want to allow me to create any new Java Free-Form Projects. It tells me “Project folder is already owned by NetBeans project my_project_name.” What is that about?

already-owned

Do any experienced NetBeans users have any suggestions on what I’m doing wrong? Is this related to Project Groups perhaps?

#ide, #intellij-idea, #netbeans

Grails File-Serving Controller and Tomcat SSL Problem

I’ve been working on a Grails application to serve up PDFs and other files, and I’ve run into a documented issue involving Internet Explorer and Apache Tomcat with SSL. I’ll explain the solution I implemented.

First of all, here’s the code I was using to serve files from a Grails controller. The list action of the controller is pretty standard – it fetches a list of documents from the database, using some selection criteria, and then it displays the documents on list.gsp, with links to the show action that pass the document’s id as a parameter.


class BulletinsController {

def list = {
List bulletins = Bulletin.getBulletins()
[params:params, bulletins:bulletins]
}

def show = {

def bulletin = bulletinService.getBulletinById(Integer.parseInt(params.id))

String fullPath =
bulletin.rootPath + "/" +
bulletin.filename
File file = new File(fullPath)
byte[] content = file.readBytes()

OutputStream out = response.getOutputStream();

// Set headers
response.setContentType(bulletin.contentType)
response.setContentLength(content.size())
response.setHeader("Content-disposition", "attachment; filename=${bulletin.filename}")

// Write the file content
out.write(content)
out.close()
}

}

This code works as expected in Firefox 3. But on Internet Explorer 7, it resulted in an error message, and no file being received:

Internet Explorer cannot download %filename% from %hostname%.

Internet Explorer cannot download filename from hostname.

Internet Explorer was not able to open this Internet site. The requested site is either unavailable or cannot be found. Please try again later.

Researching this message turned up Microsoft KB article 323308, which explains that when Internet Explorer tries to open a file using a third-party application like Adobe Reader, it has to temporarily cache the file somewhere so that the other application can access it. If the remote web server serving the file has specified Cache-control=no-cache in the HTTP response header, IE will honor this directive and will not cache the file — which causes a problem, since the third-party application won’t have a file to read.

I also found numerous mentions of a somewhat unexpected behavior of Apache Tomcat: When you define a security constraint for a resource, making it CONFIDENTIAL, Tomcat not only encrypts the resource with SSL, it also automatically appends cache control headers for it.

I confirmed that this was happening with my application by uncommenting the RequestDumperValve line in my Tomcat conf/server.xml and restarting Tomcat.

<Valve className="org.apache.catalina.valves.RequestDumperValve"/>

The next time I tried to download a PDF, Tomcat recorded the response headers in catalina.log:

Apr 21, 2009 3:36:10 PM org.apache.catalina.valves.RequestDumperValve invoke
INFO: contentLength=196923
Apr 21, 2009 3:36:10 PM org.apache.catalina.valves.RequestDumperValve invoke
INFO: contentType=application/pdf
Apr 21, 2009 3:36:10 PM org.apache.catalina.valves.RequestDumperValve invoke
INFO: header=Pragma=No-cache
Apr 21, 2009 3:36:10 PM org.apache.catalina.valves.RequestDumperValve invoke
INFO: header=Cache-Control=no-cache
Apr 21, 2009 3:36:10 PM org.apache.catalina.valves.RequestDumperValve invoke
INFO: header=Expires=Wed, 31 Dec 1969 19:00:00 EST
Apr 21, 2009 3:36:10 PM org.apache.catalina.valves.RequestDumperValve invoke
INFO: header=Content-disposition=attachment; filename=CFG-09-2012.pdf

There are ways to disable this behavior in Tomcat’s config. However, I think that this is a desirable behavior most of the time, and I’m only modifying it at all because of Internet Explorer’s problem. So my solution is to simply override the headers that Tomcat appends by appending my own in this particular Grails controller action. Here’s the working version of the show action:


def show = {
def bulletin = bulletinService.getBulletinById(Integer.parseInt(params.id))

String fullPath =
bulletin.rootPath + "/" +
bulletin.filename
File file = new File(fullPath)
byte[] content = file.readBytes()

OutputStream out = response.getOutputStream();

// Set headers
response.setContentType(bulletin.contentType)
response.setContentLength(content.size())
response.setHeader("Content-disposition", "attachment; filename=${bulletin.filename}")

/* By default, Tomcat will set headers on any SSL content to deny
* caching. This will cause downloads to Internet Explorer to fail. So
* we override Tomcat's default behavior here. */
response.setHeader("Pragma", "")
response.setHeader("Cache-Control", "private")
Calendar cal = Calendar.getInstance()
cal.add(Calendar.MINUTE,5)
response.setDateHeader("Expires", cal.getTimeInMillis())

// Write the file content
out.write(content)
out.close()
}

Setting the Pragma header to an empty string takes care of user-agents implementing HTTP 1.0. Setting the Cache-control header to private does the same for HTTP 1.1. And setting the Expires header to a value five minutes in the future instead of in the past should take care of any user-agents that use the expiration as a cache-control convention.

#cache-control, #grails, #groovy, #internet-explorer, #ssl, #tomcat

A Groovy class to generate CSV output (with tests)

I needed to write a class to output a report in comma-delimited format, and I decided to use the opportunity to practice a little Test Driven Development.

The CsvBuffer class is modeled upon StringBuffer. Instantiate one, load it with data one row at a time using append(), and then dump the output to whatever writer you want using toString() or toByteArray().

Since there is no formal specification for the comma-separated values (CSV) format, I used Wikipedia’s CSV article to define my requirements. Then I wrote a unit test:


import common.format.CsvBuffer

class CsvBufferTests extends GroovyTestCase {

String fromCodePoints(List list) {
StringBuffer buf = new StringBuffer()
list.each() {aChar ->
buf.appendCodePoint(aChar)
}
return buf.toString()
}

void testEscape() {
String neutral = fromCodePoints([115, 111, 109, 101, 32, 116, 101, 120,
116])
String containsComma = fromCodePoints([34, 116, 101, 120, 116, 44, 32,
115, 111, 109, 101, 34])
String containsDoubleQuote = fromCodePoints([34, 115, 111, 109, 101, 32,
34, 34, 116, 101, 120, 116, 34, 34, 32, 104, 101, 114, 101, 34])
String containsNewLine = fromCodePoints([34, 115, 111, 109, 101, 32,
116, 101, 120, 116, 32, 10, 104, 101, 114, 101, 34])
String containsSpaces = fromCodePoints([34, 32, 32, 115, 111, 109, 101,
32, 116, 101, 120, 116, 32, 32, 34])

CsvBuffer buf = new CsvBuffer()

/* Fields containing no special characters should not be enclosed in
* double quotes. */
assertEquals("Normal text", neutral, buf.escape("some text"))
/* Fields containing a comma should be enclosed */
assertEquals("Text containing a comma", containsComma,
buf.escape("text, some"))
/* Fields with embedded double-quote characters must be enclosed within
* double-quote characters, and each of the the embedded double-quote
* characters must be represented by a pair of double-quote
* characters. */
assertEquals("Text containing a double quote",
containsDoubleQuote, buf.escape("some \"text\" here"))
/* Fields containing a newline should be enclosed */
assertEquals("Text containing newline", containsNewLine,
buf.escape("some text \nhere"))
/* Fields with leading or trailing spaces must be enclosed within
* double-quote characters. */
assertEquals("Text containing leading or trailing spaces",
containsSpaces, buf.escape(" some text "))
}

void testAppend() {
CsvBuffer buf = new CsvBuffer()
buf.append(["String", null, 1, 2.3])
assertEquals(fromCodePoints([83, 116, 114, 105, 110, 103, 44, 44, 49,
44, 50, 46, 51, 13, 10]),
buf.toString())
}

}

With a test in hand, I then wrote a method to format output to my specification.


package common.format

class CsvBuffer {

private StringBuffer content = new StringBuffer()

/**
* Constructors. Setting column names on a header row is optional.
*/
CsvBuffer(List colnames) {
append(colnames)
}
CsvBuffer() {
// without header row
}

/**
* Returns a field value escaped for special characters
* @param input A String to be evaluated
* @return A properly formatted String
*/
String escape(String input) {
String output = new String()

if (input.contains(",") || input.contains("\n") ||
(!input.trim().equals(input))) {
output = "\"${input}\""
} else if (input.contains("\"")) {
output = "\"${input.replace("\"","\"\"")}\""
} else {
output = input
}

return output
}

/**
* Appends a row of values to the output
* @param values A list of values
* @return this CsvBuffer instance
*/
CsvBuffer append(List values) {
values.eachWithIndex() {value, i ->
// Insert a comma to delimit each field after the first
if (i > 0) {
content.append(",")
}
if (value != null) {
content.append(escape(value.toString()))
} // else null becomes ',,' - an empty string
}
content.append("\r\n")
return this
}

/**
* Outputs the contents of the buffer.
* @return Buffer contents as a String
*/
String toString() {
return content.toString()
}

/**
* Outputs the contents of the buffer.
* @return Buffer contents as a byte array
*/
byte[] toByteArray() {
return content.toString().getBytes()
}

}

I’m pretty happy with the tests for the escape() method. I’m curious – how would you recommend testing the append and output methods?

#csv, #groovy, #test-driven-development

How to sign a PDF using iText and Groovy

Paulo Soares has posted some very good examples of how to sign a PDF using iText. (iText is an open source library for working with PDFs.) I ran into a few hiccups trying to implement his example in Groovy, so I thought I’d share my fixes.

Here is a class with a method for signing PDFs. It includes the imports! This example presumes that the PDF is already encrypted to require a password for editing and that a signature field has already been placed in the PDF.

package com.geekcredential.common.output

import java.security.KeyStore
import java.security.PrivateKey
import java.security.cert.Certificate
import java.security.MessageDigest
import com.itextpdf.text.pdf.PdfReader
import com.itextpdf.text.pdf.PdfStamper
import com.itextpdf.text.pdf.PdfSignatureAppearance
import com.itextpdf.text.pdf.PdfSigGenericPKCS
import com.itextpdf.text.pdf.PdfLiteral
import com.itextpdf.text.pdf.PdfPKCS7
import com.itextpdf.text.pdf.PdfDictionary
import com.itextpdf.text.pdf.PdfString
import com.itextpdf.text.pdf.PdfName
import org.apache.log4j.Logger

/**
 * Utilities for working with PDF files.
 */

public class PdfUtil {

    /**
     * @param String keyFile (filepath to PKCS12 certificate)
     * @param String keyfilePassword
     * @param String inFile (filepath to PDF to be signed)
     * @param String ownerPassword (password for changing encrypted PDF)
     * @param String reason (text of Reason field in signature)
     * @param String location (text of Location field in signature)
     * @param String sigFieldName (name of signature field in PDF)
     * @param String outFile (desired name of signed file)
     */
    static String sign(
        String keyFile,
        String keyFilePassword,
        String inFile,
        String ownerPassword,
        String reason,
        String location,
        String sigFieldName,
        String outFile
    ) {
        Logger log = Logger.getLogger("com.geekcredential.common.output.PdfUtil")
        log.debug("Entering sign()")
        try {
            // Load the certificate to be used for signing
            KeyStore ks = KeyStore.getInstance("pkcs12")
            ks.load(new FileInputStream(keyFile), keyFilePassword.toCharArray())
            // Defaults here to first alias found in keyfile. We could change this
            // to specify a named alias.
            String alias = (String)ks.aliases().nextElement()
            PrivateKey key = (PrivateKey)ks.getKey(alias, keyFilePassword.toCharArray())
            Certificate[] chain = ks.getCertificateChain(alias)

            PdfReader reader = new PdfReader(inFile, ownerPassword.getBytes())
            FileOutputStream fout = new FileOutputStream(outFile)
            // Third param is PDF revision (char). Fifth param (boolean) enables append without
            // incrementing revision number.
            // Groovy thinks '\0' is a GString, so we have to be explicit and force it to char.
            PdfStamper stp = PdfStamper.createSignature(reader, fout, '\0'.toCharacter().charValue(), null, true)
            PdfSignatureAppearance sap = stp.getSignatureAppearance()
            sap.setCrypto(key, chain, null, PdfSignatureAppearance.SELF_SIGNED)
            sap.setReason(reason)
            sap.setLocation(location)
            sap.setVisibleSignature(sigFieldName)
            stp.close()
            return outFile
        } catch (Throwable t) {
            log.fatal("Failure signing PDF", t)
            throw t
        } finally {
            log.debug("Exiting sign()")
        }
    }
}

Note: iText supports three certificate signing modes: SELF_SIGNED, WINCER_SIGNED and VERISIGN_SIGNED. I could find no trace of the Verisign Adobe Reader plugin that is supposed to require VERISIGN_SIGNED on Verisign’s website, so I presume that this is old technology, discontinued. However, our Verisign test certificate works just fine with WINCER_SIGNED, and as Paulo Soares points out, WINCER_SIGNED mode works with any certificate. So, in practice, it seems that there are really only two modes: SELF_SIGNED or WINCER_SIGNED.

#digital-signature, #groovy, #itext, #pdf

Validating intent

<g:actionSubmit action="sign" value="I agree"
    title="I agree" tabindex="-1"
    style="position: relative;
        left: ${Math.floor(Math.random() * 600).intValue()}px;"/>

Do I just have an evil sense of humor? No. What makes an electronic signature legally valid is the conscious intent of the signer to sign. This is the submit button I’m using on a form to collect an electronic signature. It would be very difficult to click this button by accident. The random positioning also prevents many automated tools from clicking it. If a customer ever wants to contest that they have willingly accepted the terms of this electronic signature, I’ve just made it more difficult for them to prove their case in court.

Still, I can’t help getting an evil chuckle out of writing this code.

#electronic-signature, #gsp

Convert XML to Groovy

Having discovered that I couldn’t simply embed a bunch of XSL-FO templates into my Groovy classes as Strings, I’m looking for a way to convert XSL-FO to Groovy MarkupBuilder script.

There are lots of articles about how to use Groovy to write or consume XML. There are not so many about how to convert in the opposite direction. Codehaus.org gives a brief example in Creating XML using Groovy’s MarkupBuilder of how to use the DomToGroovy class.

Based on that example, I created a class with a method for converting XML to Groovy and a script to call it from the command line.

Unfortunately, DomToGroovy doesn’t seem to handle XSL-FO so well. Still this will be a handy tool to keep around.

ConvertXml.groovy:

package com.geekcredential.common.fileformat.xml
/**
* Based on http://groovy.codehaus.org/Creating+XML+using+Groovy's+MarkupBuilder
*/
import org.codehaus.groovy.tools.xml.DomToGroovy
import javax.xml.parsers.DocumentBuilderFactory
import org.apache.log4j.Logger

public class ConvertXml {

static Logger log = new Logger("com.geekcredential.common.fileformat.xml.ConvertXml")

static void toGroovy(String sourcePath, String targetPath) {
try {
File sourceFile = new File(sourcePath)
File targetFile = new File(targetPath)

def builder = DocumentBuilderFactory.newInstance().newDocumentBuilder()
def inputStream = new ByteArrayInputStream(sourceFile.readBytes())
def document = builder.parse(inputStream)
def output = new StringWriter()
def converter = new DomToGroovy(new PrintWriter(output))

converter.print(document)
targetFile.withWriter {fw ->
fw << output.toString()
}
} catch (Throwable t) {
log.fatal("Error converting XML file to Groovy script", t)
}
}
}

xmlToGroovy.groovy:

import com.geekcredential.common.fileformat.xml.ConvertXml

def targetFilePath = new String()

if (args.size() 2) {
println "Usage: groovy xmlToGroovy.groovy sourceFilePath [targetFilePath]"
println "(If targetFilePath is not specified, output will be written to"
println "the same name as sourceFilePath, but with a .groovy extension.)"
System.exit(1)
}

def sourceFilePath = args[0]

if (args.size() == 1) {
targetFilePath = sourceFilePath.substring(0, sourceFilePath.lastIndexOf(".")) + ".groovy"
} else {
targetFilePath = args[1]
}

ConvertXml.toGroovy(sourceFilePath.toString(), targetFilePath.toString())

System.exit(0)

#domtogroovy, #groovy, #xml, #xsl-fo