Uploading large files

[Ep]

I want to create a content upload service that sha1 hashes the content and stores it to a compressed file. Some files may be quite large (2GB+ uncompressed). Would unfiltered be a good fit for implementing this type of service?

[n8han]

Yes, assuming you intend to use http. Is the remote client a regular browser or something custom?

Nathan

On 07/21/2011 04:15 PM, Ep [via Databinder] wrote:

I want to create a content upload service that sha1 hashes the content and stores it to a compressed file. Some files may be quite large (2GB+ uncompressed). Would unfiltered be a good fit for implementing this type of service?

---

If you reply to this email, your message will be added to the discussion below:

http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608364.html

To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.

[Dustin Whitney]

I've been curious about this myself.  I'd like to hear what your experience is.

Dustin

On Thu, Jul 21, 2011 at 4:17 PM, n8han [via Databinder] <[hidden email]> wrote:

Yes, assuming you intend to use http. Is the remote client a regular browser or something custom?

Nathan

On 07/21/2011 04:15 PM, Ep [via Databinder] wrote:

I want to create a content upload service that sha1 hashes the content and stores it to a compressed file. Some files may be quite large (2GB+ uncompressed). Would unfiltered be a good fit for implementing this type of service?

---

If you reply to this email, your message will be added to the discussion below:

http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608364.html

To start a new topic under Unfiltered, email [hidden email]

To unsubscribe from Unfiltered, click here.

---

If you reply to this email, your message will be added to the discussion below:

http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608375.html

To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.

[Ep]

Something custom. Basically something like SillyStore but the key is a sha1 of the bytes...and save/load the compressed files to/from disk. How to sha1 and compress as content is streamed to disk? How to decompress and stream to client? etc.

[chris_lewis]

Now you're outside the realm of unfiltered, but ...

Hashes (like sha) are based on the full contents of a file, so you can't resolve its hash without its full contents. You could investigate calculating hashes incrementally. That is, for every 100mb, calculate a sha and aggregate the hashes per each chunk. If your goal is to generate the expected <name>.sha, then this option is out.

I'm not sure about the compression part, but I imagine you can't do it, at least not well, incrementally. I'd be in input from others though!

-chris

On 7/21/11 4:31 PM, Ep [via Databinder] wrote:

Something custom. Basically something like SillyStore but the key is a sha1 of the bytes...and save/load the compressed files to/from disk. How to sha1 and compress as content is streamed to disk? How to decompress and stream to client? etc.

---

If you reply to this email, your message will be added to the discussion below:

http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608417.html

To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.

[chris_lewis]

On 7/21/11 4:38 PM, Chris Lewis wrote:

Now you're outside the realm of unfiltered, but ...

Hashes (like sha) are based on the full contents of a file, so you can't resolve its hash without its full contents. You could investigate calculating hashes incrementally. That is, for every 100mb, calculate a sha and aggregate the hashes per each chunk. If your goal is to generate the expected <name>.sha, then this option is out.

I'm not sure about the compression part, but I imagine you can't do it, at least not well, incrementally. I'd be in input from others though!

Ahem - "interested in."

-chris

On 7/21/11 4:31 PM, Ep [via Databinder] wrote:

Something custom. Basically something like SillyStore but the key is a sha1 of the bytes...and save/load the compressed files to/from disk. How to sha1 and compress as content is streamed to disk? How to decompress and stream to client? etc.

---

If you reply to this email, your message will be added to the discussion below:

http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608417.html

To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.

[Ep]

May just hash then compress the file on the client (or could do it on the server as well once the file is uploaded). So it then becomes...how to move large file via http with unfiltered and what are the size limits?

[n8han]

You are going to have any problem with size limits from unfiltered. You can choose to handle a blocking InputStream coming from a filter, or the non-blocking interfaces that Netty gives you.

If you intend to support many simultaneous uploads and don't need to work through the servlet API, Netty is probably a better choice. But it's worth building prototypes of each.

Nathan

On 07/21/2011 04:43 PM, Ep [via Databinder] wrote:

May just hash then compress the file on the client (or could do it on the server as well once the file is uploaded). So it then becomes...how to move large file via http with unfiltered and what are the size limits?

---

If you reply to this email, your message will be added to the discussion below:

http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608458.html

To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.

[Ep]

Not restricted to the servlet api and will give Netty a try. But with the servlet route, would you recommend the upload plugin?

[n8han]

Not really. That plugin handles multipart posts and since you're in control of the client you can do a simple PUT like the silly store example, with less overhead and complexity.

On 07/21/2011 05:05 PM, Ep [via Databinder] wrote:

Not restricted to the servlet api and will give Netty a try. But with the servlet route, would you recommend the upload plugin?

---

If you reply to this email, your message will be added to the discussion below:

http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608516.html

To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.

[Dustin Whitney]

If you use Jetty, you'll need to set the post size limit higher.  Here is how I do it - not sure if anyone has a better way:

for(hs <- server.underlying.getHandlers) hs match {
      case handlers:ContextHandlerCollection =>
        for(h <- handlers.getHandlers) h match {
          case s:ServletContextHandler =>
            s.setMaxFormContentSize(5242880) //5mb limit
        }
    }

On Thu, Jul 21, 2011 at 5:13 PM, n8han [via Databinder] <[hidden email]> wrote:

Not really. That plugin handles multipart posts and since you're in control of the client you can do a simple PUT like the silly store example, with less overhead and complexity.

On 07/21/2011 05:05 PM, Ep [via Databinder] wrote:

Not restricted to the servlet api and will give Netty a try. But with the servlet route, would you recommend the upload plugin?

---

If you reply to this email, your message will be added to the discussion below:

http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608516.html

To start a new topic under Unfiltered, email [hidden email]

To unsubscribe from Unfiltered, click here.

---

If you reply to this email, your message will be added to the discussion below:

http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608540.html

To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.

[n8han]

I don't think that should apply to the request body, cause that would be really lame! I think it is just if you are trying to get it out of a parameter, if you're using a multipart post.

http://permalink.gmane.org/gmane.comp.java.jetty.support/13510

Nathan

On 07/22/2011 09:32 AM, Dustin Whitney [via Databinder] wrote:

If you use Jetty, you'll need to set the post size limit higher.  Here is how I do it - not sure if anyone has a better way:

for(hs <- server.underlying.getHandlers) hs match {
      case handlers:ContextHandlerCollection =>
        for(h <- handlers.getHandlers) h match {
          case s:ServletContextHandler =>
            s.setMaxFormContentSize(5242880) //5mb limit
        }
    }

On Thu, Jul 21, 2011 at 5:13 PM, n8han [via Databinder] <[hidden email]> wrote:

Not really. That plugin handles multipart posts and since you're in control of the client you can do a simple PUT like the silly store example, with less overhead and complexity.

On 07/21/2011 05:05 PM, Ep [via Databinder] wrote:

Not restricted to the servlet api and will give Netty a try. But with the servlet route, would you recommend the upload plugin?

---

If you reply to this email, your message will be added to the discussion below:

http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608516.html

To start a new topic under Unfiltered, email [hidden email]

To unsubscribe from Unfiltered, click here.

---

If you reply to this email, your message will be added to the discussion below:

http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608540.html

To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.

---

If you reply to this email, your message will be added to the discussion below:

http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6610475.html

To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.