Post by "JuanFeng", 2006-10-18, 13:10
-----------------------------------------------------
问题背景: 为了解决资产的大数据量流式传输问题, 我作了一个HTTP的数据传输通道, SERVER端把对象二进制序列化后, 发二进制包到CLIENT, CLIENT收到到包后,再反序列化为对象! 使用HTTP的原因,是因为HTTP支持SERVER端在处理数据的同时,可以把已经处理的部分数据先发回CLIENT端, 然后继续处理其他数据,以加快客户端的响应!
问题现象: 这样的方式在我自己的开发PC上(OS:2003)一直工作的很好, 但是目前在测试人员的PC上(OS:WINXP)出现如下问题:
Ø 偶尔,CLIENT端在从网络流中接收数据包时,总是收到0字节(请见下面代码中的红色部分),所以CLIENT端就不得不循环接收这个包,最终导致CLIENT数据接收不完整!
Ø 但是,此时CLIENT端的网络流没有中断, SERVER端发数据包的情况也正常!
Ø 这种情况不会在某个环境上稳定再现,我在另外一个开发用的XP操作系统上,不能复现这个问题,
Ø 每次出现问题的时机也不相同, 比如: 第一次错误是出现在接收第1300个对象时, 而第二个出错点可能是在接收2000个对象等等.
我的解决方案:
Ø 由于此前的工作模式是,CLIENT接收一个(或一批)对象就导出它们,但是在导出的同时, 就暂时不接收数据, 等导出完成了,再继续接收,直到处理完所有对象!
Ø 我怀疑这种方式可能会造成网络的缓冲压力太大, 因为当客户端不接收的时候, SERVER端并不知道,所以继续给CLIENT发数据包;
Ø 为此,我想把CLIENT端的数据导出做成了异步的, 保证主线程可以不顿地接收数据包;
Ø CLIENT改造成异步以后, 情况有所改观, 但依然不能避免问题的出现!!
主要的CODE 大致如下:
Code of SERVER side(Page_Load Of a aspx page):
Dim responce as stream = Me.Context.Response.OutputStream
dim obj as object
obj = ????? ' 一个需要传递到CLIENT的业务对象
Dim bnSer As New Runtime.Serialization.Formatters.Binary.BinaryFormatter
Dim iSize As Integer
Dim stream As New IO.MemoryStream
bnSer.Serialize(stream, obj)
Dim buff() As Byte = stream.GetBuffer
iSize = buff.Length
If iSize <= 0 Then
stream.Close()
stream = Nothing
bnSer = Nothing
responce.Close
Return False
End If
responce .Write(BitConverter.GetBytes(iSize), 0, 4)
responce .Write(buff, 0, iSize)
responce .Flush()
responce.Close
stream.Close()
stream = Nothing
bnSer = Nothing
Code of CLIENT side:
Dim request As System.Net.HttpWebRequest = System.Net.WebRequest.Create(Me._Url)
request.ContentLength=0
Dim response As System.Net.HttpWebResponse = request.GetResponse
Dim st As IO.Stream = response.GetResponseStream
Dim stReceive As New IO.BufferedStream(st)
Dim bnSer As New Runtime.Serialization.Formatters.Binary.BinaryFormatter
Dim iSize As Integer = 4
Dim iCount As Integer = iSize
Dim buff(iSize - 1) As Byte
Dim len As Integer = -1
Dim iRealSize As Int64 = 0
While iRealSize < iSize
'如果一次不能接收所有的数据,则下次继续接收,直到接收完整(iRealSize = iSize)
iRealSize = iRealSize + stReceive.Read(buff, iRealSize, iSize - iRealSize)
End While
len = BitConverter.ToInt32(buff, 0)
Dim objBuff(len - 1) As Byte
iRealSize = 0
While iRealSize < len
'如果一次不能接收所有的数据,则下次继续接收,直到接收完整(iRealSize = len)
iRealSize = iRealSize + stReceive.Read(objBuff, iRealSize, len - iRealSize)
End While
Dim stream As New IO.MemoryStream(objBuff)
Dim obj As Object
obj = bnSer.Deserialize(stream)
至此,CLIENT获得了obj, 其他代码省略
Reply by "Alfred",2006-10-18, 14:37
-----------------------------------------------------
先要记录一些Log，以供分析这个问题，比如返回的Response Stream的长度，等等的参数。
现在这个处理的环节比较多，比如 Binary 序列化，Aspx 文件的 Render，HTTP 协议，HTTPClient 自动的处理等等，经过这么多环节后，我们很难判断出是哪一个环节出了问题。
那么就用一些办法，从头开始尝试，比如刚开始的时候，仅仅使用常量的BinaryArray发送和接收数据，确定网络传输上没有问题，再增加上 Binary 序列化，再次尝试，依次类推，一次比一次使用更多的环节，从而找出问题。
我已经写了一个简单的程序实现楼主所说的传输机制，从简单的测试上看，没有问题，稳定性也不错，稍后我会追加更多的环节。见附件。
Reply by "JuanFeng", 2006-10-18, 15:49
-----------------------------------------------------
各位, 关于这个问题,又有新发现:
我在XP的DEBUG状态下,偶然跟踪到下面一个EXCEPTION,
Probable I/O race condition detected while copying memory. The I/O package is not thread safe by default. In multithreaded applications, a stream must be accessed in a thread-safe way, such as a thread-safe wrapper returned by TextReader's or TextWriter's Synchronized methods. This also applies to classes like StreamWriter and StreamReader.
Reply by "grapeman", 2006-10-18, 17:27
-----------------------------------------------------
FYI
How to do Synchronous and Asynchronous web downloads
http://www.codeproject.com/managedcpp/cswget01.asp
Reply by "grapeman", 2006-10-18, 17:51
-----------------------------------------------------
有人也再现了这样的问题，并成功解决了。
http://ryangregg.com/default,date,2004-10-26.aspx
Lately I've been encountering random crashes of an application I was working on, and I couldn't figure out the cause. It's a C# program that writes information to a log file during a work cycle so that I can get a good debug log of everything that goes on.
However, randomly during a write operation I would get various unhandled exceptions, with vague messages about how Count needs to be greater than 0 and less than the buffer length, and other related messages. I didn't have a clue what was occurring, but I figured it might have something to do with the way all the work was being carrier out on a separate thread, and that this potentially might be a sign of the TextWriter not support multi-threaded access.
Finally one of the crashes proved this was the case, when I got this exception:
An unhandled exception of type 'System.IndexOutOfRangeException' occurred in mscorlib.dll
Additional information: Probable I/O race condition detected while copying memory. The I/O package is not thread safe by default. In multithreaded applications, a stream must be accessed in a thread-safe way, such as a thread-safe wrapper returned by TextReader's or TextWriter's Synchronized methods. This also applies to classes like StreamWriter and StreamReader.
Finally a reason for my problems! Using the thread-safe wrapper class returned by the Synchroized method solved the problem completely, and I haven't seen it hang up on me once since then.
StreamWriter originalWriter = new StreamWriter(LocalStore.CreateFile(TANK_LOG_FILE));
// Make sure we have a thread-safe writer to use here
writer = StreamWriter.Synchronized(originalWriter);
The interesting twist here is that all of the stream writing was taking place on a single thread, so I'm not sure why this condition presented itself. Perhaps further analysis of the code is necessary to see if there is a condition in which multiple writes could be occurring at the same time. A good indication that something was wrong though was that my data was being scrambled at the end of the file, intermixed with data from other parts of the log output (making the log file in general very useless). All of this has cleared up now thanks to that helpful error message, and a little bit of intelligence in the code.
Reply by "St.Valentine",2006-10-18, 18:00
-----------------------------------------------------
抛砖引玉一下：）
上文中一个重要的元素：System.Threading.ManualResetEvent配合AutoResetEvent实现对线程“运行状态”的维护。
//莫非上文的例子只是要告诉我们Thread的执行顺序和状态可以靠这两个家伙来控制？感觉有些乱，等待达人：）
--下面zz from msdn
Notifies one or more waiting threads that an event has occurred. This class cannot be inherited.
Namespace: System.Threading
Assembly: mscorlib (in mscorlib.dll)
Syntax
Visual Basic (Declaration)
<ComVisibleAttribute(True)> _
Public NotInheritable Class ManualResetEvent
Inherits EventWaitHandle
Visual Basic (Usage)
Dim instance As ManualResetEvent
C#
[ComVisibleAttribute(true)]
public sealed class ManualResetEvent : EventWaitHandle
C++
[ComVisibleAttribute(true)]
public ref class ManualResetEvent sealed : public EventWaitHandle
J#
/** @attribute ComVisibleAttribute(true) */
public final class ManualResetEvent extends EventWaitHandle
JScript
ComVisibleAttribute(true)
public final class ManualResetEvent extends EventWaitHandle
Remarks
Note
The HostProtectionAttribute attribute applied to this class has the following Resources property value: Synchronization | ExternalThreading. The HostProtectionAttribute does not affect desktop applications (which are typically started by double-clicking an icon, typing a command, or entering a URL in a browser). For more information, see the HostProtectionAttribute class or SQL Server Programming and Host Protection Attributes.
In the .NET Framework version 2.0, ManualResetEvent derives from the new EventWaitHandle class. A ManualResetEvent is functionally equivalent to an EventWaitHandle created with EventResetMode.ManualReset.
Note
Unlike the ManualResetEvent class, the EventWaitHandle class provides access to named system synchronization events.
ManualResetEvent allows threads to communicate with each other by signaling. Typically, this communication concerns a task which one thread must complete before other threads can proceed.
When a thread begins an activity that must complete before other threads proceed, it calls Reset to put ManualResetEvent in the non-signaled state. This thread can be thought of as controlling the ManualResetEvent. Threads that call WaitOne on the ManualResetEvent will block, awaiting the signal. When the controlling thread completes the activity, it calls Set to signal that the waiting threads can proceed. All waiting threads are released.
Once it has been signaled, ManualResetEvent remains signaled until it is manually reset. That is, calls to WaitOne return immediately.
You can control the initial state of a ManualResetEvent by passing a Boolean value to the constructor, true if the initial state is signaled and false otherwise.
ManualResetEvent can also be used with the staticWaitAll and WaitAny methods.
For more information about thread synchronization mechanisms, see ManualResetEvent in the conceptual documentation.
Example
The following code example demonstrates how to use wait handles to signal the completion of various stages of a complicated number calculation. The calculation is of the form: result = first term + second term + third term, where each term requires a precalculation and a final calculation using a calculated base number.
Visual Basic Copy Code
Imports System
Imports System.Threading
Public Class CalculateTest
<MTAThreadAttribute> _
Shared Sub Main()
Dim calc As New Calculate()
Console.WriteLine("Result = {0}.", _
calc.Result(234).ToString())
Console.WriteLine("Result = {0}.", _
calc.Result(55).ToString())
End Sub
End Class
Public Class Calculate
Dim baseNumber, firstTerm, secondTerm, thirdTerm As Double
Dim autoEvents() As AutoResetEvent
Dim manualEvent As ManualResetEvent
' Generate random numbers to simulate the actual calculations.
Dim randomGenerator As Random
Sub New()
autoEvents = New AutoResetEvent(2) { _
New AutoResetEvent(False), _
New AutoResetEvent(False), _
New AutoResetEvent(False) }
manualEvent = New ManualResetEvent(False)
End Sub
Private Sub CalculateBase(stateInfo As Object)
baseNumber = randomGenerator.NextDouble()
' Signal that baseNumber is ready.
manualEvent.Set()
End Sub
' The following CalculateX methods all perform the same
' series of steps as commented in CalculateFirstTerm.
Private Sub CalculateFirstTerm(stateInfo As Object)
' Perform a precalculation.
Dim preCalc As Double = randomGenerator.NextDouble()
' Wait for baseNumber to be calculated.
manualEvent.WaitOne()
' Calculate the first term from preCalc and baseNumber.
firstTerm = preCalc * baseNumber * _
randomGenerator.NextDouble()
' Signal that the calculation is finished.
autoEvents(0).Set()
End Sub
Private Sub CalculateSecondTerm(stateInfo As Object)
Dim preCalc As Double = randomGenerator.NextDouble()
manualEvent.WaitOne()
secondTerm = preCalc * baseNumber * _
randomGenerator.NextDouble()
autoEvents(1).Set()
End Sub
Private Sub CalculateThirdTerm(stateInfo As Object)
Dim preCalc As Double = randomGenerator.NextDouble()
manualEvent.WaitOne()
thirdTerm = preCalc * baseNumber * _
randomGenerator.NextDouble()
autoEvents(2).Set()
End Sub
Function Result(seed As Integer) As Double
randomGenerator = New Random(seed)
' Simultaneously calculate the terms.
ThreadPool.QueueUserWorkItem(AddressOf CalculateBase)
ThreadPool.QueueUserWorkItem(AddressOf CalculateFirstTerm)
ThreadPool.QueueUserWorkItem(AddressOf CalculateSecondTerm)
ThreadPool.QueueUserWorkItem(AddressOf CalculateThirdTerm)
' Wait for all of the terms to be calculated.
WaitHandle.WaitAll(autoEvents)
' Reset the wait handle for the next calculation.
manualEvent.Reset()
Return firstTerm + secondTerm + thirdTerm
End Function
End Class
C# Copy Code
using System;
using System.Threading;
class CalculateTest
{
static void Main()
{
Calculate calc = new Calculate();
Console.WriteLine("Result = {0}.",
calc.Result(234).ToString());
Console.WriteLine("Result = {0}.",
calc.Result(55).ToString());
}
}
class Calculate
{
double baseNumber, firstTerm, secondTerm, thirdTerm;
AutoResetEvent[] autoEvents;
ManualResetEvent manualEvent;
// Generate random numbers to simulate the actual calculations.
Random randomGenerator;
public Calculate()
{
autoEvents = new AutoResetEvent[]
{
new AutoResetEvent(false),
new AutoResetEvent(false),
new AutoResetEvent(false)
};
manualEvent = new ManualResetEvent(false);
}
void CalculateBase(object stateInfo)
{
baseNumber = randomGenerator.NextDouble();
// Signal that baseNumber is ready.
manualEvent.Set();
}
// The following CalculateX methods all perform the same
// series of steps as commented in CalculateFirstTerm.
void CalculateFirstTerm(object stateInfo)
{
// Perform a precalculation.
double preCalc = randomGenerator.NextDouble();
// Wait for baseNumber to be calculated.
manualEvent.WaitOne();
// Calculate the first term from preCalc and baseNumber.
firstTerm = preCalc * baseNumber *
randomGenerator.NextDouble();
// Signal that the calculation is finished.
autoEvents[0].Set();
}
void CalculateSecondTerm(object stateInfo)
{
double preCalc = randomGenerator.NextDouble();
manualEvent.WaitOne();
secondTerm = preCalc * baseNumber *
randomGenerator.NextDouble();
autoEvents[1].Set();
}
void CalculateThirdTerm(object stateInfo)
{
double preCalc = randomGenerator.NextDouble();
manualEvent.WaitOne();
thirdTerm = preCalc * baseNumber *
randomGenerator.NextDouble();
autoEvents[2].Set();
}
public double Result(int seed)
{
randomGenerator = new Random(seed);
// Simultaneously calculate the terms.
ThreadPool.QueueUserWorkItem(
new WaitCallback(CalculateBase));
ThreadPool.QueueUserWorkItem(
new WaitCallback(CalculateFirstTerm));
ThreadPool.QueueUserWorkItem(
new WaitCallback(CalculateSecondTerm));
ThreadPool.QueueUserWorkItem(
new WaitCallback(CalculateThirdTerm));
// Wait for all of the terms to be calculated.
WaitHandle.WaitAll(autoEvents);
// Reset the wait handle for the next calculation.
manualEvent.Reset();
return firstTerm + secondTerm + thirdTerm;
}
}
Reply by "JuanFeng",2006-10-19, 11:17
-----------------------------------------------------
我修改了一下, 在SERVER端把bytes转换为chars, 传递到CLIENT, CLIENT对chars解析!!!
但是出现了以下错误 我们通过PING查看网络状态, 出错时,网络状态正常!)
{System.ObjectDisposedException}
[System.ObjectDisposedException]: {System.ObjectDisposedException}
HelpLink: Nothing
InnerException: Nothing
Message: "[Resource lookup failed - infinite recursion detected. Resource name: ObjectDisposed_Generic_ObjectName]
[Resource lookup failed - infinite recursion detected. Resource name: ObjectDisposed_ObjectName_Name]"
Source: "System"
StackTrace: " at System.Net.ConnectStream.BeginRead(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.StreamReader.ReadBuffer(Char[] userBuffer, Int32 userOffset, Int32 desiredChars, Boolean& readToUserBuffer)
at System.IO.StreamReader.Read(Char[] buffer, Int32 index, Int32 count)
at System.IO.SyncTextReader.Read(Char[] buffer, Int32 index, Int32 count)
at LeySer.Client.Shisan.WebNewActionProvider_RD.GetAssetObject(Object _query, Type _type, String TaskID, Boolean MultThread) in D:\SHISAN8_Free\Client\LeySer.Client.Shisan.Core\System\WebNewActionProvider_RD.vb:line 403"
TargetSite: {System.Reflection.RuntimeMethodInfo}
Reply by "dujid", 2006-10-20, 16:35
-----------------------------------------------------
char应该不行， char是 -127 ~ 127，如果是 二进制序列化一定不行吧。
Reply by " JuanFeng", 2006-10-20, 16:56
-----------------------------------------------------
由于我们不能有效地解决这个问题, 加之,时间紧迫, 我们已经把HTTP传递对象的部分功能, 迁移到SOAP, 使用WEBSERVER接收由SERVER端的二进制对象, 绕开了这个问题!
希望感兴趣的朋友,继续研究这个问题, 如果有结果,请告诉大家!
非常感谢以上各位朋友,对我们的大力支持!!!
冯军海 |
|
陕公网安备：61019002000258 |
隐私政策 |
网站地图